Trace number 3321743

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-random? (MO) 7883.1 998.916

General information on the benchmark

Namerandom/large/
unif-k3-r4.2-v50000-c210000-S1969540674-065-UNKNOWN.cnf
MD5SUMfcb14103324f734a2e69e02c062e67ed
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark1079.13
Satisfiable
(Un)Satisfiability was proved
Number of variables50000
Number of clauses210000
Sum of the clauses size630000
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 3210000
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.31/0.46	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.31/0.46	c INFO: solving HOME/instance-3321743-1304366567.cnf
0.31/0.46	c INFO: computing task features
1.10/1.20	c INFO: computed variable-clause graph statistics
1.10/1.23	c INFO: computed clause balance statistics
4.70/4.81	c INFO: computed variable balance statistics
7.51/7.68	c INFO: computed small-clause counts
7.51/7.70	c INFO: computed Horn-clause counts
10.60/10.77	c INFO: computed variable graph statistics
10.60/10.77	c INFO: collected features for HOME/instance-3321743-1304366567.cnf in 10.29 s
10.70/10.81	c INFO: running hybridGM3@0 for 97 with 39989 remaining (b = 0.35)
10.70/10.85	c INFO: running hybridGM3@0 for 97 with 39989 remaining (b = 0.21)
10.70/10.89	c INFO: running hybridGM3@0 for 97 with 39989 remaining (b = 0.14)
10.91/10.93	c INFO: running hybridGM3@0 for 194 with 39989 remaining (b = 0.18)
10.91/10.97	c INFO: running hybridGM3@0 for 194 with 39989 remaining (b = 0.13)
11.21/11.01	c INFO: running hybridGM3@0 for 194 with 39989 remaining (b = 0.10)
11.21/11.05	c INFO: running hybridGM3@0 for 194 with 39989 remaining (b = 0.07)
11.21/11.09	c INFO: running TNM@0 for 97 with 39989 remaining (b = 0.03)
810.42/111.18	c INFO: running hybridGM3@0 for 97 with 39889 remaining (b = 0.03)
811.09/111.22	c INFO: running hybridGM3@0 for 97 with 39889 remaining (b = 0.03)
811.09/111.26	c INFO: running hybridGM3@0 for 97 with 39889 remaining (b = 0.03)
811.80/111.37	c INFO: running hybridGM3@0 for 194 with 39889 remaining (b = 0.05)
1570.61/206.41	c INFO: running hybridGM3@100 for 97 with 39794 remaining (b = 0.02)
1570.61/206.46	c INFO: running hybridGM3@100 for 97 with 39794 remaining (b = 0.02)
1571.39/206.50	c INFO: running hybridGM3@99 for 97 with 39794 remaining (b = 0.02)
1578.57/207.49	c INFO: running hybridGM3@0 for 97 with 39793 remaining (b = 0.02)
1579.34/207.57	c INFO: running hybridGM3@0 for 97 with 39792 remaining (b = 0.02)
1580.15/207.63	c INFO: running hybridGM3@0 for 97 with 39792 remaining (b = 0.02)
1627.21/213.58	c INFO: running hybridGM3@0 for 97 with 39786 remaining (b = 0.02)
2339.09/302.71	c INFO: running hybridGM3@0 for 194 with 39697 remaining (b = 0.04)
2378.19/307.66	c INFO: running hybridGM3@0 for 194 with 39692 remaining (b = 0.03)
2378.95/307.71	c INFO: running hybridGM3@0 for 194 with 39692 remaining (b = 0.03)
2378.95/307.77	c INFO: running hybridGM3@296 for 97 with 39692 remaining (b = 0.01)
2380.56/308.00	c INFO: running TNM@0 for 97 with 39692 remaining (b = 0.01)
2386.87/308.83	c INFO: running hybridGM3@95 for 97 with 39691 remaining (b = 0.01)
2386.87/308.88	c INFO: running hybridGM3@95 for 97 with 39691 remaining (b = 0.01)
2387.48/308.93	c INFO: running TNM@99 for 1940 with 39691 remaining (b = 0.20)
3139.32/403.02	c INFO: running hybridGM3@0 for 194 with 39597 remaining (b = 0.02)
3148.08/404.13	c INFO: running hybridGM3@0 for 194 with 39596 remaining (b = 0.02)
3181.59/408.35	c INFO: running adaptg2wsat2009++@0 for 194 with 39592 remaining (b = 0.02)
3195.13/410.10	c INFO: running hybridGM3@189 for 97 with 39590 remaining (b = 0.01)
3907.06/499.38	c INFO: running hybridGM3@100 for 97 with 39501 remaining (b = 0.01)
3946.02/504.30	c INFO: running hybridGM3@101 for 97 with 39496 remaining (b = 0.01)
3954.76/505.35	c INFO: running hybridGM3@386 for 97 with 39495 remaining (b = 0.01)
3994.66/510.40	c INFO: running hybridGM3@100 for 97 with 39490 remaining (b = 0.01)
4666.56/594.64	c INFO: running hybridGM3@297 for 97 with 39405 remaining (b = 0.01)
4707.15/599.72	c INFO: running hybridGM3@95 for 97 with 39400 remaining (b = 0.01)
4714.38/600.61	c INFO: running TNM@0 for 97 with 39399 remaining (b = 0.01)
4715.01/600.79	c INFO: running hybridGM3@98 for 97 with 39399 remaining (b = 0.01)
4739.84/603.87	c INFO: running TNM@0 for 97 with 39396 remaining (b = 0.01)
4753.31/605.57	c INFO: running hybridGM3@0 for 194 with 39394 remaining (b = 0.02)
4754.07/605.66	c INFO: running TNM@0 for 97 with 39394 remaining (b = 0.01)
5473.80/695.93	c INFO: running TNM@0 for 97 with 39304 remaining (b = 0.01)
5474.51/696.08	c INFO: running hybridGM3@293 for 194 with 39304 remaining (b = 0.02)
5513.59/700.93	c INFO: running TNM@0 for 97 with 39299 remaining (b = 0.01)
5514.33/701.03	c INFO: running hybridGM3@190 for 97 with 39299 remaining (b = 0.01)
5539.06/704.19	c INFO: running TNM@0 for 97 with 39296 remaining (b = 0.01)
5544.75/705.00	c INFO: running adaptg2wsat2009++@100 for 97 with 39295 remaining (b = 0.01)
6265.12/795.33	c INFO: running TNM@0 for 97 with 39205 remaining (b = 0.01)
6273.05/796.33	c INFO: running TNM@0 for 97 with 39204 remaining (b = 0.01)
6312.00/801.28	c INFO: running TNM@0 for 97 with 39199 remaining (b = 0.01)
6312.78/801.39	c INFO: running adaptg2wsat2009++@0 for 194 with 39199 remaining (b = 0.01)
6320.70/802.31	c INFO: running hybridGM3@94 for 97 with 39198 remaining (b = 0.01)
6338.21/804.52	c INFO: running hybridGM3@0 for 194 with 39196 remaining (b = 0.01)
7063.36/895.68	c INFO: running TNM@0 for 97 with 39104 remaining (b = 0.01)
7071.30/896.68	c INFO: running hybridGM3@195 for 291 with 39103 remaining (b = 0.02)
7079.29/897.63	c INFO: running TNM@0 for 97 with 39102 remaining (b = 0.01)
7095.11/899.63	c INFO: running hybridGM3@390 for 97 with 39100 remaining (b = 0.01)
7111.00/901.64	c INFO: running TNM@0 for 97 with 39098 remaining (b = 0.01)
7861.51/996.05	c INFO: running TNM@0 for 97 with 39004 remaining (b = 0.01)
7869.45/997.03	c INFO: running TNM@0 for 97 with 39003 remaining (b = 0.01)
7876.58/998.00	c INFO: running TNM@0 for 97 with 39002 remaining (b = 0.01)
7883.10/998.87	c INFO: running hybridGM3@195 for 291 with 39001 remaining (b = 0.02)
7883.10/998.87	Traceback (most recent call last):
7883.10/998.87	  File "HOME/solve", line 33, in <module>
7883.10/998.87	    plac.call(main)
7883.10/998.87	  File "/home/evaluation/evaluation/tools64/lib/python2.7/site-packages/plac-0.8.1-py2.7.egg/plac_core.py", line 311, in call
7883.10/998.87	    cmd, result = parser_from(obj).consume(arglist)
7883.10/998.87	  File "/home/evaluation/evaluation/tools64/lib/python2.7/site-packages/plac-0.8.1-py2.7.egg/plac_core.py", line 192, in consume
7883.10/998.87	    return cmd, self.func(*(args + varargs + extraopts), **kwargs)
7883.10/998.87	  File "HOME/borg/build/lib/borg/tools/solve.py", line 84, in main
7883.10/998.87	    (_, answer, _) = solver(input_path, budget, cores)
7883.10/998.87	  File "HOME/borg/build/lib/borg/bilevel.py", line 202, in __call__
7883.10/998.87	    solver.go(max_cost * borg.defaults.machine_speed)
7883.10/998.87	  File "HOME/borg/build/lib/borg/solvers.py", line 179, in go
7883.10/998.87	    self._process.start()
7883.10/998.87	  File "/home/evaluation/evaluation/tools64/lib/python2.7/multiprocessing/process.py", line 94, in start
7883.10/998.87	    assert self._popen is None, 'cannot start a process twice'
7883.10/998.87	AssertionError: cannot start a process twice

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3321743-1304366567/watcher-3321743-1304366567 -o /tmp/evaluation-result-3321743-1304366567/solver-3321743-1304366567 -C 40000 -W 5100 -M 31000 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3321743-1304366567.cnf -seed 1719099220 -budget 40000 -cores 8 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 4.06 1.25 0.45 2/269 29038
/proc/meminfo: memFree=23867944/32951124 swapFree=67111464/67111528
[pid=29038] ppid=29036 vsize=15772 CPUtime=0 cores=0-7
/proc/29038/stat : 29038 (python) R 29036 29038 28808 0 -1 4202496 895 0 0 0 0 0 0 0 21 0 1 0 548254468 16150528 776 33554432000 4194304 5683644 140736016732768 18446744073709551615 212687675024 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/29038/statm: 3943 776 343 364 0 430 0

[startup+0.079024 s]
/proc/loadavg: 4.06 1.25 0.45 2/269 29038
/proc/meminfo: memFree=23867944/32951124 swapFree=67111464/67111528
[pid=29038] ppid=29036 vsize=85360 CPUtime=0.05 cores=0-7
/proc/29038/stat : 29038 (python) R 29036 29038 28808 0 -1 4202496 2593 0 0 0 4 1 0 0 22 0 1 0 548254468 87408640 2314 33554432000 4194304 5683644 140736016732768 18446744073709551615 212687673344 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/29038/statm: 21340 2314 713 364 0 1557 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 85360

[startup+0.101007 s]
/proc/loadavg: 4.06 1.25 0.45 2/269 29038
/proc/meminfo: memFree=23867944/32951124 swapFree=67111464/67111528
[pid=29038] ppid=29036 vsize=104224 CPUtime=0.07 cores=0-7
/proc/29038/stat : 29038 (python) R 29036 29038 28808 0 -1 4202496 3071 0 0 0 5 2 0 0 22 0 1 0 548254468 106725376 2751 33554432000 4194304 5683644 140736016732768 18446744073709551615 212687673344 0 2147483391 16781312 2 0 0 0 17 6 0 0 0
/proc/29038/statm: 26056 2751 847 364 0 1804 0
Current children cumulated CPU time (s) 0.07
Current children cumulated vsize (KiB) 104224

[startup+0.300992 s]
/proc/loadavg: 4.06 1.25 0.45 2/269 29038
/proc/meminfo: memFree=23867944/32951124 swapFree=67111464/67111528
[pid=29038] ppid=29036 vsize=172852 CPUtime=0.22 cores=0-7
/proc/29038/stat : 29038 (python) S 29036 29038 28808 0 -1 4202496 6744 932 1 0 15 7 0 0 17 0 1 0 548254468 177000448 4812 33554432000 4194304 5683644 140736016732768 18446744073709551615 212682751623 0 2147483391 16781312 2 18446744071699199166 0 0 17 6 0 0 0
/proc/29038/statm: 43213 4812 1338 364 0 3452 0
Current children cumulated CPU time (s) 0.22
Current children cumulated vsize (KiB) 172852

[startup+0.700898 s]
/proc/loadavg: 4.06 1.25 0.45 2/269 29038
/proc/meminfo: memFree=23867944/32951124 swapFree=67111464/67111528
[pid=29038] ppid=29036 vsize=297044 CPUtime=0.59 cores=0-7
/proc/29038/stat : 29038 (python) R 29036 29038 28808 0 -1 4202496 13863 932 1 0 50 9 0 0 17 0 1 0 548254468 304173056 11548 33554432000 4194304 5683644 140736016732768 18446744073709551615 47236445625111 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/29038/statm: 74261 11548 1761 364 0 10185 0
Current children cumulated CPU time (s) 0.59
Current children cumulated vsize (KiB) 297044

[startup+1.50074 s]
/proc/loadavg: 4.06 1.25 0.45 2/270 29043
/proc/meminfo: memFree=23810896/32951124 swapFree=67111464/67111528
[pid=29038] ppid=29036 vsize=327996 CPUtime=1.4 cores=0-7
/proc/29038/stat : 29038 (python) R 29036 29038 28808 0 -1 4202496 27450 932 1 0 129 11 0 0 19 0 1 0 548254468 335867904 19384 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699512512 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/29038/statm: 81999 19384 1826 364 0 17923 0
Current children cumulated CPU time (s) 1.4
Current children cumulated vsize (KiB) 327996

[startup+3.10045 s]
/proc/loadavg: 4.06 1.25 0.45 2/270 29043
/proc/meminfo: memFree=23797132/32951124 swapFree=67111464/67111528
[pid=29038] ppid=29036 vsize=327996 CPUtime=3 cores=0-7
/proc/29038/stat : 29038 (python) R 29036 29038 28808 0 -1 4202496 27450 932 1 0 289 11 0 0 22 0 1 0 548254468 335867904 19384 33554432000 4194304 5683644 140736016732768 18446744073709551615 5003632 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/29038/statm: 81999 19384 1826 364 0 17923 0
Current children cumulated CPU time (s) 3
Current children cumulated vsize (KiB) 327996

[startup+6.30087 s]
/proc/loadavg: 3.98 1.28 0.46 2/270 29043
/proc/meminfo: memFree=23797948/32951124 swapFree=67111464/67111528
[pid=29038] ppid=29036 vsize=327996 CPUtime=6.2 cores=0-7
/proc/29038/stat : 29038 (python) R 29036 29038 28808 0 -1 4202496 27451 932 1 0 609 11 0 0 25 0 1 0 548254468 335867904 19385 33554432000 4194304 5683644 140736016732768 18446744073709551615 4510864 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/29038/statm: 81999 19385 1827 364 0 17923 0
Current children cumulated CPU time (s) 6.2
Current children cumulated vsize (KiB) 327996

[startup+12.7017 s]
/proc/loadavg: 3.74 1.28 0.46 9/294 29068
/proc/meminfo: memFree=23614756/32951124 swapFree=67111464/67111528
[pid=29038] ppid=29036 vsize=406152 CPUtime=10.99 cores=0-7
/proc/29038/stat : 29038 (python) S 29036 29038 28808 0 -1 4202496 34278 932 1 0 1085 14 0 0 25 0 9 0 548254468 415899648 18503 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699502027 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/29038/statm: 101538 18503 1889 364 0 37435 0
[pid=29045] ppid=29038 vsize=324116 CPUtime=0 cores=0-7
/proc/29045/stat : 29045 (python) S 29038 29038 28808 0 -1 4202560 734 0 0 0 0 0 0 0 18 0 1 0 548255550 331894784 16935 33554432000 4194304 5683644 140736016732768 18446744073709551615 212687695375 0 0 16781312 514 1078409255154876415 0 0 17 1 0 0 0
/proc/29045/statm: 81029 16935 353 364 0 16947 0
[pid=29047] ppid=29045 vsize=276236 CPUtime=1.87 cores=0-7
/proc/29047/stat : 29047 (hybridGM3) R 29045 29047 29047 0 -1 4202496 5151 0 0 0 186 1 0 0 25 0 1 0 548255550 282865664 4885 33554432000 134512640 135204385 4292977696 18446744073709551615 134543975 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/29047/statm: 69059 4885 57 169 0 68889 0
[pid=29048] ppid=29038 vsize=334372 CPUtime=0 cores=0-7
/proc/29048/stat : 29048 (python) S 29038 29038 28808 0 -1 4202560 730 0 0 0 0 0 0 0 18 0 1 0 548255554 342396928 16933 33554432000 4194304 5683644 140736016732768 18446744073709551615 212687695430 0 0 16781312 514 4522763019825971199 0 0 17 4 0 0 0
/proc/29048/statm: 83593 16933 347 364 0 19508 0
[pid=29050] ppid=29048 vsize=276236 CPUtime=1.83 cores=0-7
/proc/29050/stat : 29050 (hybridGM3) R 29048 29050 29050 0 -1 4202496 5114 0 0 0 182 1 0 0 25 0 1 0 548255554 282865664 4850 33554432000 134512640 135204385 4289925552 18446744073709551615 134543975 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/29050/statm: 69059 4850 57 169 0 68889 0
[pid=29051] ppid=29038 vsize=344628 CPUtime=0 cores=0-7
/proc/29051/stat : 29051 (python) S 29038 29038 28808 0 -1 4202560 738 0 0 0 0 0 0 0 18 0 1 0 548255558 352899072 16937 33554432000 4194304 5683644 140736016732768 18446744073709551615 212687695430 0 0 16781312 514 415293242687356927 0 0 17 1 0 0 0
/proc/29051/statm: 86157 16937 347 364 0 22069 0
[pid=29053] ppid=29051 vsize=276236 CPUtime=1.79 cores=0-7
/proc/29053/stat : 29053 (hybridGM3) R 29051 29053 29053 0 -1 4202496 5094 0 0 0 178 1 0 0 25 0 1 0 548255558 282865664 4832 33554432000 134512640 135204385 4290085488 18446744073709551615 134543975 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/29053/statm: 69059 4832 57 169 0 68889 0
[pid=29054] ppid=29038 vsize=354884 CPUtime=0 cores=0-7
/proc/29054/stat : 29054 (python) S 29038 29038 28808 0 -1 4202560 751 0 0 0 0 0 0 0 18 0 1 0 548255561 363401216 16941 33554432000 4194304 5683644 140736016732768 18446744073709551615 212687695430 0 0 16781312 514 4522766868116668415 0 0 17 0 0 0 0
/proc/29054/statm: 88721 16941 347 364 0 24630 0
[pid=29056] ppid=29054 vsize=276232 CPUtime=1.75 cores=0-7
/proc/29056/stat : 29056 (hybridGM3) R 29054 29056 29056 0 -1 4202496 5057 0 0 0 175 0 0 0 25 0 1 0 548255562 282861568 4795 33554432000 134512640 135204385 4290429104 18446744073709551615 134543975 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/29056/statm: 69058 4795 57 169 0 68888 0
[pid=29057] ppid=29038 vsize=365140 CPUtime=0 cores=0-7
/proc/29057/stat : 29057 (python) S 29038 29038 28808 0 -1 4202560 757 0 0 0 0 0 0 0 18 0 1 0 548255566 373903360 16945 33554432000 4194304 5683644 140736016732768 18446744073709551615 212687695430 0 0 16781312 514 4522759171535273983 0 0 17 1 0 0 0
/proc/29057/statm: 91285 16945 347 364 0 27191 0
[pid=29059] ppid=29057 vsize=276232 CPUtime=1.69 cores=0-7
/proc/29059/stat : 29059 (hybridGM3) R 29057 29059 29059 0 -1 4202496 5023 0 0 0 169 0 0 0 25 0 1 0 548255566 282861568 4758 33554432000 134512640 135204385 4293067984 18446744073709551615 134543968 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/29059/statm: 69058 4758 57 169 0 68888 0
[pid=29060] ppid=29038 vsize=375396 CPUtime=0 cores=0-7
/proc/29060/stat : 29060 (python) S 29038 29038 28808 0 -1 4202560 759 0 0 0 0 0 0 0 18 0 1 0 548255569 384405504 16949 33554432000 4194304 5683644 140736016732768 18446744073709551615 212687695430 0 0 16781312 514 4522770716407365631 0 0 17 6 0 0 0
/proc/29060/statm: 93849 16949 347 364 0 29752 0
[pid=29062] ppid=29060 vsize=276232 CPUtime=1.67 cores=0-7
/proc/29062/stat : 29062 (hybridGM3) R 29060 29062 29062 0 -1 4202496 5005 0 0 0 166 1 0 0 25 0 1 0 548255570 282861568 4743 33554432000 134512640 135204385 4286864288 18446744073709551615 134543975 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/29062/statm: 69058 4743 57 169 0 68888 0
[pid=29063] ppid=29038 vsize=385652 CPUtime=0 cores=0-7
/proc/29063/stat : 29063 (python) S 29038 29038 28808 0 -1 4202560 771 0 0 0 0 0 0 0 18 0 1 0 548255574 394907648 16953 33554432000 4194304 5683644 140736016732768 18446744073709551615 212687695430 0 0 16781312 514 6520376361012428799 0 0 17 0 0 0 0
/proc/29063/statm: 96413 16953 347 364 0 32313 0
[pid=29065] ppid=29063 vsize=276232 CPUtime=1.62 cores=0-7
/proc/29065/stat : 29065 (hybridGM3) R 29063 29065 29065 0 -1 4202496 4969 0 0 0 161 1 0 0 25 0 1 0 548255574 282861568 4700 33554432000 134512640 135204385 4293873552 18446744073709551615 134543993 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/29065/statm: 69058 4700 57 169 0 68888 0
[pid=29066] ppid=29038 vsize=395908 CPUtime=0 cores=0-7
/proc/29066/stat : 29066 (python) S 29038 29038 28808 0 -1 4202560 796 0 0 0 0 0 0 0 18 0 1 0 548255578 405409792 16963 33554432000 4194304 5683644 140736016732768 18446744073709551615 212687695430 0 0 16781312 514 4606905071286878207 0 0 17 6 0 0 0
/proc/29066/statm: 98977 16963 353 364 0 34874 0
[pid=29068] ppid=29066 vsize=471116 CPUtime=1.56 cores=0-7
/proc/29068/stat : 29068 (TNM) R 29066 29068 29068 0 -1 4202496 8436 0 0 0 156 0 0 0 25 0 1 0 548255578 482422784 8170 33554432000 134512640 135008748 4289017088 18446744073709551615 134527773 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/29068/statm: 117779 8170 46 122 0 116679 0
[pid=29038/tid=29046] ppid=29036 vsize=406152 CPUtime=0 cores=0-7
/proc/29038/task/29046/stat : 29046 (python) S 29036 29038 28808 0 -1 4202560 27 932 0 0 0 0 0 0 25 0 9 0 548255550 415899648 18503 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699498929 0 0 16781312 2 18446604440278878976 0 0 -1 0 0 0 0
[pid=29038/tid=29049] ppid=29036 vsize=406152 CPUtime=0 cores=0-7
/proc/29038/task/29049/stat : 29049 (python) S 29036 29038 28808 0 -1 4202560 23 932 0 0 0 0 0 0 25 0 9 0 548255554 415899648 18503 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699498929 0 0 16781312 2 18446604440278878976 0 0 -1 0 0 0 0
[pid=29038/tid=29052] ppid=29036 vsize=406152 CPUtime=0 cores=0-7
/proc/29038/task/29052/stat : 29052 (python) S 29036 29038 28808 0 -1 4202560 23 932 0 0 0 0 0 0 19 0 9 0 548255558 415899648 18503 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699498929 0 0 16781312 2 18446604440278878976 0 0 -1 0 0 0 0
[pid=29038/tid=29055] ppid=29036 vsize=406152 CPUtime=0 cores=0-7
/proc/29038/task/29055/stat : 29055 (python) S 29036 29038 28808 0 -1 4202560 24 932 0 0 0 0 0 0 20 0 9 0 548255562 415899648 18503 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699498929 0 0 16781312 2 18446604440278878976 0 0 -1 4 0 0 0
[pid=29038/tid=29058] ppid=29036 vsize=406152 CPUtime=0 cores=0-7
/proc/29038/task/29058/stat : 29058 (python) S 29036 29038 28808 0 -1 4202560 25 932 0 0 0 0 0 0 19 0 9 0 548255566 415899648 18503 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699498929 0 0 16781312 2 18446604440278878976 0 0 -1 0 0 0 0
[pid=29038/tid=29061] ppid=29036 vsize=406152 CPUtime=0 cores=0-7
/proc/29038/task/29061/stat : 29061 (python) S 29036 29038 28808 0 -1 4202560 28 932 0 0 0 0 0 0 21 0 9 0 548255570 415899648 18503 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699498929 0 0 16781312 2 18446604440278878976 0 0 -1 6 0 0 0
[pid=29038/tid=29064] ppid=29036 vsize=406152 CPUtime=0 cores=0-7
/proc/29038/task/29064/stat : 29064 (python) S 29036 29038 28808 0 -1 4202560 26 932 0 0 0 0 0 0 23 0 9 0 548255574 415899648 18503 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699498929 0 0 16781312 2 18446604440278878976 0 0 -1 0 0 0 0
[pid=29038/tid=29067] ppid=29036 vsize=406152 CPUtime=0 cores=0-7
/proc/29038/task/29067/stat : 29067 (python) S 29036 29038 28808 0 -1 4202560 32 932 0 0 0 0 0 0 25 0 9 0 548255578 415899648 18503 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699498929 0 0 16781312 2 18446604440278878976 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 24.77
Current children cumulated vsize (KiB) 5691000

[startup+25.5013 s]
/proc/loadavg: 4.68 1.61 0.58 9/294 29068
/proc/meminfo: memFree=23474360/32951124 swapFree=67111464/67111528
[pid=29038] ppid=29036 vsize=406152 CPUtime=10.99 cores=0-7
/proc/29038/stat : 29038 (python) S 29036 29038 28808 0 -1 4202496 34278 932 1 0 1085 14 0 0 25 0 9 0 548254468 415899648 18503 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699502027 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/29038/statm: 101538 18503 1889 364 0 37435 0
[pid=29045] ppid=29038 vsize=324116 CPUtime=0.02 cores=0-7
/proc/29045/stat : 29045 (python) S 29038 29038 28808 0 -1 4202560 908 0 0 0 1 1 0 0 15 0 1 0 548255550 331894784 16958 33554432000 4194304 5683644 140736016732768 18446744073709551615 212687695375 0 0 16781312 514 745134632097251588 0 0 17 1 0 0 0
/proc/29045/statm: 81029 16958 376 364 0 16947 0
[pid=29047] ppid=29045 vsize=276240 CPUtime=14.66 cores=0-7
/proc/29047/stat : 29047 (hybridGM3) R 29045 29047 29047 0 -1 4202496 9516 0 0 0 1465 1 0 0 25 0 1 0 548255550 282869760 9250 33554432000 134512640 135204385 4292977696 18446744073709551615 134537483 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/29047/statm: 69060 9250 64 169 0 68890 0
[pid=29048] ppid=29038 vsize=334372 CPUtime=0.02 cores=0-7
/proc/29048/stat : 29048 (python) S 29038 29038 28808 0 -1 4202560 905 0 0 0 1 1 0 0 15 0 1 0 548255554 342396928 16956 33554432000 4194304 5683644 140736016732768 18446744073709551615 212687695430 0 0 16781312 514 745134632097251585 0 0 17 4 0 0 0
/proc/29048/statm: 83593 16956 370 364 0 19508 0
[pid=29050] ppid=29048 vsize=276240 CPUtime=14.63 cores=0-7
/proc/29050/stat : 29050 (hybridGM3) R 29048 29050 29050 0 -1 4202496 9517 0 0 0 1462 1 0 0 25 0 1 0 548255554 282869760 9253 33554432000 134512640 135204385 4289925552 18446744073709551615 134536605 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/29050/statm: 69060 9253 64 169 0 68890 0
[pid=29051] ppid=29038 vsize=344628 CPUtime=0.02 cores=0-7
/proc/29051/stat : 29051 (python) S 29038 29038 28808 0 -1 4202560 927 0 0 0 1 1 0 0 15 0 1 0 548255558 352899072 16960 33554432000 4194304 5683644 140736016732768 18446744073709551615 212687695430 0 0 16781312 514 745134632097251587 0 0 17 1 0 0 0
/proc/29051/statm: 86157 16960 370 364 0 22069 0
[pid=29053] ppid=29051 vsize=276240 CPUtime=14.59 cores=0-7
/proc/29053/stat : 29053 (hybridGM3) R 29051 29053 29053 0 -1 4202496 9515 0 0 0 1458 1 0 0 25 0 1 0 548255558 282869760 9253 33554432000 134512640 135204385 4290085488 18446744073709551615 134537573 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/29053/statm: 69060 9253 64 169 0 68890 0
[pid=29054] ppid=29038 vsize=354884 CPUtime=0.01 cores=0-7
/proc/29054/stat : 29054 (python) S 29038 29038 28808 0 -1 4202560 941 0 0 0 0 1 0 0 15 0 1 0 548255561 363401216 16964 33554432000 4194304 5683644 140736016732768 18446744073709551615 212687695430 0 0 16781312 514 745134632097251588 0 0 17 0 0 0 0
/proc/29054/statm: 88721 16964 370 364 0 24630 0
[pid=29056] ppid=29054 vsize=276236 CPUtime=14.53 cores=0-7
/proc/29056/stat : 29056 (hybridGM3) R 29054 29056 29056 0 -1 4202496 9512 0 0 0 1452 1 0 0 25 0 1 0 548255562 282865664 9250 33554432000 134512640 135204385 4290429104 18446744073709551615 134546153 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/29056/statm: 69059 9250 64 169 0 68889 0
[pid=29057] ppid=29038 vsize=365140 CPUtime=0.02 cores=0-7
/proc/29057/stat : 29057 (python) S 29038 29038 28808 0 -1 4202560 951 0 0 0 1 1 0 0 15 0 1 0 548255566 373903360 16968 33554432000 4194304 5683644 140736016732768 18446744073709551615 212687695430 0 0 16781312 514 745134632097251585 0 0 17 1 0 0 0
/proc/29057/statm: 91285 16968 370 364 0 27191 0
[pid=29059] ppid=29057 vsize=276236 CPUtime=14.35 cores=0-7
/proc/29059/stat : 29059 (hybridGM3) R 29057 29059 29059 0 -1 4202496 9528 0 0 0 1434 1 0 0 25 0 1 0 548255566 282865664 9263 33554432000 134512640 135204385 4293067984 18446744073709551615 134545866 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/29059/statm: 69059 9263 64 169 0 68889 0

################
# More data... #
################

[pid=29150] ppid=29038 vsize=615284 CPUtime=0.23 cores=0-7
/proc/29150/stat : 29150 (python) S 29038 29038 28808 0 -1 4202560 1188 0 0 0 13 10 0 0 19 0 2 0 548324887 630050816 17365 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699502027 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/29150/statm: 153821 17365 407 364 0 89673 0
[pid=29152] ppid=29150 vsize=471116 CPUtime=100.14 cores=0-7
/proc/29152/stat : 29152 (TNM) T 29150 29152 29152 0 -1 4202496 8453 0 0 0 10013 1 0 0 25 0 1 0 548324888 482422784 8172 33554432000 134512640 135008748 4291284096 18446744073709551615 134527320 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/29152/statm: 117779 8172 46 122 0 116679 0
[pid=29150/tid=29172] ppid=29038 vsize=615284 CPUtime=0 cores=0-7
/proc/29150/task/29172/stat : 29172 (python) S 29038 29038 28808 0 -1 4202560 106 0 0 0 0 0 0 0 19 0 2 0 548334911 630050816 17365 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699498929 0 0 16781312 514 18446604444348723456 0 0 -1 4 0 0 0
[pid=29157] ppid=29038 vsize=625540 CPUtime=0.23 cores=0-7
/proc/29157/stat : 29157 (python) S 29038 29038 28808 0 -1 4202560 1198 0 0 0 11 12 0 0 19 0 2 0 548334001 640552960 17353 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699502027 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/29157/statm: 156385 17353 406 364 0 92234 0
[pid=29159] ppid=29157 vsize=471116 CPUtime=100.17 cores=0-7
/proc/29159/stat : 29159 (TNM) T 29157 29159 29159 0 -1 4202496 8455 0 0 0 10015 2 0 0 25 0 1 0 548334002 482422784 8171 33554432000 134512640 135008748 4294402832 18446744073709551615 134527795 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/29159/statm: 117779 8171 46 122 0 116679 0
[pid=29157/tid=29178] ppid=29038 vsize=625540 CPUtime=0 cores=0-7
/proc/29157/task/29178/stat : 29178 (python) S 29038 29038 28808 0 -1 4202560 106 0 0 0 0 0 0 0 19 0 2 0 548344025 640552960 17353 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699498929 0 0 16781312 514 18446604453952183104 0 0 -1 5 0 0 0
[pid=29160] ppid=29038 vsize=635796 CPUtime=0.23 cores=0-7
/proc/29160/stat : 29160 (python) S 29038 29038 28808 0 -1 4202560 1229 0 0 0 14 9 0 0 19 0 2 0 548334102 651055104 17304 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699502027 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/29160/statm: 158949 17304 400 364 0 94795 0
[pid=29162] ppid=29160 vsize=471116 CPUtime=100.21 cores=0-7
/proc/29162/stat : 29162 (TNM) T 29160 29162 29162 0 -1 4202496 8456 0 0 0 10019 2 0 0 25 0 1 0 548334102 482422784 8170 33554432000 134512640 135008748 4292508512 18446744073709551615 134527693 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/29162/statm: 117779 8170 46 122 0 116679 0
[pid=29160/tid=29182] ppid=29038 vsize=635796 CPUtime=0 cores=0-7
/proc/29160/task/29182/stat : 29182 (python) S 29038 29038 28808 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 548344126 651055104 17304 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699498929 0 0 16781312 514 18446604451038380224 0 0 -1 4 0 0 0
[pid=29165] ppid=29038 vsize=646052 CPUtime=0.24 cores=0-7
/proc/29165/stat : 29165 (python) S 29038 29038 28808 0 -1 4202560 1268 0 0 0 13 11 0 0 19 0 2 0 548334596 661557248 17308 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699502027 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/29165/statm: 161513 17308 400 364 0 97356 0
[pid=29167] ppid=29165 vsize=471116 CPUtime=99.24 cores=0-7
/proc/29167/stat : 29167 (TNM) T 29165 29167 29167 0 -1 4202496 8454 0 0 0 9922 2 0 0 25 0 1 0 548334598 482422784 8169 33554432000 134512640 135008748 4289199088 18446744073709551615 134524687 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/29167/statm: 117779 8169 46 122 0 116679 0
[pid=29165/tid=29186] ppid=29038 vsize=646052 CPUtime=0 cores=0-7
/proc/29165/task/29186/stat : 29186 (python) S 29038 29038 28808 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 548344621 661557248 17308 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699498929 0 0 16781312 514 18446604453942683712 0 0 -1 0 0 0 0
[pid=29168] ppid=29038 vsize=656308 CPUtime=0.48 cores=0-7
/proc/29168/stat : 29168 (python) S 29038 29038 28808 0 -1 4202560 1292 0 0 0 23 25 0 0 19 0 2 0 548334608 672059392 17365 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699502027 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/29168/statm: 164077 17365 400 364 0 99917 0
[pid=29170] ppid=29168 vsize=438604 CPUtime=195.06 cores=0-7
/proc/29170/stat : 29170 (adaptg2wsat2009) T 29168 29170 29170 0 -1 4202496 8776 0 0 0 19505 1 0 0 25 0 1 0 548334608 449130496 8497 33554432000 134512640 135107411 4290846224 18446744073709551615 134531935 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/29170/statm: 109651 8497 59 146 0 108527 0
[pid=29168/tid=29197] ppid=29038 vsize=656308 CPUtime=0 cores=0-7
/proc/29168/task/29197/stat : 29197 (python) S 29038 29038 28808 0 -1 4202560 215 0 0 0 0 0 0 0 19 0 2 0 548354160 672059392 17365 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699498929 0 0 16781312 514 18446604451038382016 0 0 -1 5 0 0 0
[pid=29173] ppid=29038 vsize=666564 CPUtime=0.46 cores=0-7
/proc/29173/stat : 29173 (python) S 29038 29038 28808 0 -1 4202560 984 0 0 0 21 25 0 0 15 0 1 0 548334921 682561536 17504 33554432000 4194304 5683644 140736016732768 18446744073709551615 212687695430 0 0 16781312 514 745134632097251585 0 0 17 2 0 0 0
/proc/29173/statm: 166641 17504 373 364 0 102478 0
[pid=29175] ppid=29173 vsize=276244 CPUtime=193.42 cores=0-7
/proc/29175/stat : 29175 (hybridGM3) R 29173 29175 29175 0 -1 4202496 9542 0 0 0 19339 3 0 0 25 0 1 0 548334921 282873856 9258 33554432000 134512640 135204385 4293118480 18446744073709551615 134537292 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/29175/statm: 69061 9258 64 169 0 68891 0
[pid=29179] ppid=29038 vsize=676820 CPUtime=0.25 cores=0-7
/proc/29179/stat : 29179 (python) S 29038 29038 28808 0 -1 4202560 1229 0 0 0 13 12 0 0 19 0 2 0 548344037 693063680 17493 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699502027 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/29179/statm: 169205 17493 406 364 0 105039 0
[pid=29181] ppid=29179 vsize=471116 CPUtime=99.96 cores=0-7
/proc/29181/stat : 29181 (TNM) T 29179 29181 29181 0 -1 4202496 8457 0 0 0 9995 1 0 0 25 0 1 0 548344037 482422784 8170 33554432000 134512640 135008748 4286788208 18446744073709551615 134527542 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/29181/statm: 117779 8170 46 122 0 116679 0
[pid=29179/tid=29193] ppid=29038 vsize=676820 CPUtime=0 cores=0-7
/proc/29179/task/29193/stat : 29193 (python) S 29038 29038 28808 0 -1 4202560 93 0 0 0 0 0 0 0 19 0 2 0 548354062 693063680 17493 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699498929 0 0 16781312 514 18446604449043936384 0 0 -1 4 0 0 0
[pid=29183] ppid=29038 vsize=687076 CPUtime=0.25 cores=0-7
/proc/29183/stat : 29183 (python) S 29038 29038 28808 0 -1 4202560 1235 0 0 0 13 12 0 0 19 0 2 0 548344232 703565824 17474 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699502027 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/29183/statm: 171769 17474 407 364 0 107600 0
[pid=29185] ppid=29183 vsize=471116 CPUtime=100.21 cores=0-7
/proc/29185/stat : 29185 (TNM) T 29183 29185 29185 0 -1 4202496 8458 0 0 0 10020 1 0 0 25 0 1 0 548344232 482422784 8170 33554432000 134512640 135008748 4289806688 18446744073709551615 134529409 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/29185/statm: 117779 8170 46 122 0 116679 0
[pid=29183/tid=29201] ppid=29038 vsize=687076 CPUtime=0 cores=0-7
/proc/29183/task/29201/stat : 29201 (python) S 29038 29038 28808 0 -1 4202560 100 0 0 0 0 0 0 0 19 0 2 0 548354257 703565824 17474 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699498929 0 0 16781312 514 18446604444163212352 0 0 -1 1 0 0 0
[pid=29187] ppid=29038 vsize=697332 CPUtime=0.23 cores=0-7
/proc/29187/stat : 29187 (python) S 29038 29038 28808 0 -1 4202560 1042 0 0 0 12 11 0 0 18 0 1 0 548344632 714067968 17453 33554432000 4194304 5683644 140736016732768 18446744073709551615 212687695430 0 0 16781312 514 745134632097251586 0 0 17 2 0 0 0
/proc/29187/statm: 174333 17453 378 364 0 110161 0
[pid=29189] ppid=29187 vsize=471116 CPUtime=95.86 cores=0-7
/proc/29189/stat : 29189 (TNM) R 29187 29189 29189 0 -1 4202496 8459 0 0 0 9585 1 0 0 25 0 1 0 548344634 482422784 8171 33554432000 134512640 135008748 4288444528 18446744073709551615 134527118 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/29189/statm: 117779 8171 46 122 0 116679 0
[pid=29194] ppid=29038 vsize=707588 CPUtime=0 cores=0-7
/proc/29194/stat : 29194 (python) S 29038 29038 28808 0 -1 4202560 952 0 0 0 0 0 0 0 18 0 1 0 548354073 724570112 17528 33554432000 4194304 5683644 140736016732768 18446744073709551615 212687695430 0 0 16781312 514 7249284125629055235 0 0 17 2 0 0 0
/proc/29194/statm: 176897 17528 355 364 0 112722 0
[pid=29196] ppid=29194 vsize=471116 CPUtime=2.64 cores=0-7
/proc/29196/stat : 29196 (TNM) R 29194 29196 29196 0 -1 4202496 8460 0 0 0 263 1 0 0 25 0 1 0 548354074 482422784 8170 33554432000 134512640 135008748 4294706464 18446744073709551615 134529367 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/29196/statm: 117779 8170 46 122 0 116679 0
[pid=29198] ppid=29038 vsize=717844 CPUtime=0 cores=0-7
/proc/29198/stat : 29198 (python) S 29038 29038 28808 0 -1 4202560 927 0 0 0 0 0 0 0 18 0 1 0 548354172 735072256 17590 33554432000 4194304 5683644 140736016732768 18446744073709551615 212687695430 0 0 16781312 514 4875289262557233151 0 0 17 2 0 0 0
/proc/29198/statm: 179461 17590 349 364 0 115283 0
[pid=29200] ppid=29198 vsize=471116 CPUtime=1.66 cores=0-7
/proc/29200/stat : 29200 (TNM) R 29198 29200 29200 0 -1 4202496 8464 0 0 0 165 1 0 0 25 0 1 0 548354173 482422784 8172 33554432000 134512640 135008748 4294058160 18446744073709551615 134527267 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/29200/statm: 117779 8172 46 122 0 116679 0
[pid=29202] ppid=29038 vsize=728100 CPUtime=0 cores=0-7
/proc/29202/stat : 29202 (python) S 29038 29038 28808 0 -1 4202560 943 0 0 0 0 0 0 0 19 0 1 0 548354269 745574400 17404 33554432000 4194304 5683644 140736016732768 18446744073709551615 212687695430 0 0 16781312 514 4875296959138627583 0 0 17 2 0 0 0
/proc/29202/statm: 182025 17404 345 364 0 117844 0
[pid=29204] ppid=29202 vsize=465836 CPUtime=0.7 cores=0-7
/proc/29204/stat : 29204 (TNM) R 29202 29204 29204 0 -1 4202496 5317 0 0 0 70 0 0 0 22 0 1 0 548354269 477016064 5025 33554432000 134512640 135008748 4290141584 18446744073709551615 134580187 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/29204/statm: 116459 5025 46 122 0 115359 0
[pid=29038/tid=29046] ppid=29036 vsize=738344 CPUtime=0 cores=0-7
/proc/29038/task/29046/stat : 29046 (python) S 29036 29038 28808 0 -1 4202560 36 932 0 0 0 0 0 0 18 0 35 0 548255550 756064256 19005 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699498929 0 0 16781312 2 18446604440278878976 0 0 -1 0 0 0 0
[pid=29038/tid=29049] ppid=29036 vsize=738344 CPUtime=0 cores=0-7
/proc/29038/task/29049/stat : 29049 (python) S 29036 29038 28808 0 -1 4202560 31 932 0 0 0 0 0 0 15 0 35 0 548255554 756064256 19005 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699498929 0 0 16781312 2 18446604440278878976 0 0 -1 0 0 0 0
[pid=29038/tid=29052] ppid=29036 vsize=738344 CPUtime=0 cores=0-7
/proc/29038/task/29052/stat : 29052 (python) S 29036 29038 28808 0 -1 4202560 32 932 0 0 0 0 0 0 18 0 35 0 548255558 756064256 19005 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699498929 0 0 16781312 2 18446604440278878976 0 0 -1 6 0 0 0
[pid=29038/tid=29055] ppid=29036 vsize=738344 CPUtime=0 cores=0-7
/proc/29038/task/29055/stat : 29055 (python) S 29036 29038 28808 0 -1 4202560 36 932 0 0 0 0 0 0 18 0 35 0 548255562 756064256 19005 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699498929 0 0 16781312 2 18446604440278878976 0 0 -1 4 0 0 0
[pid=29038/tid=29058] ppid=29036 vsize=738344 CPUtime=0 cores=0-7
/proc/29038/task/29058/stat : 29058 (python) S 29036 29038 28808 0 -1 4202560 32 932 0 0 0 0 0 0 18 0 35 0 548255566 756064256 19005 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699498929 0 0 16781312 2 18446604440278878976 0 0 -1 0 0 0 0
[pid=29038/tid=29061] ppid=29036 vsize=738344 CPUtime=0 cores=0-7
/proc/29038/task/29061/stat : 29061 (python) S 29036 29038 28808 0 -1 4202560 33 932 0 0 0 0 0 0 18 0 35 0 548255570 756064256 19005 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699498929 0 0 16781312 2 18446604440278878976 0 0 -1 6 0 0 0
[pid=29038/tid=29064] ppid=29036 vsize=738344 CPUtime=0 cores=0-7
/proc/29038/task/29064/stat : 29064 (python) S 29036 29038 28808 0 -1 4202560 30 932 0 0 0 0 0 0 18 0 35 0 548255574 756064256 19005 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699498929 0 0 16781312 2 18446604440278878976 0 0 -1 0 0 0 0
[pid=29038/tid=29067] ppid=29036 vsize=738344 CPUtime=0 cores=0-7
/proc/29038/task/29067/stat : 29067 (python) S 29036 29038 28808 0 -1 4202560 36 932 0 0 0 0 0 0 18 0 35 0 548255578 756064256 19005 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699498929 0 0 16781312 2 18446604440278878976 0 0 -1 4 0 0 0
[pid=29038/tid=29079] ppid=29036 vsize=738344 CPUtime=0 cores=0-7
/proc/29038/task/29079/stat : 29079 (python) S 29036 29038 28808 0 -1 4202560 34 932 0 0 0 0 0 0 15 0 35 0 548265606 756064256 19005 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699498929 0 0 16781312 2 18446604440278878976 0 0 -1 3 0 0 0
[pid=29038/tid=29091] ppid=29036 vsize=738344 CPUtime=0 cores=0-7
/proc/29038/task/29091/stat : 29091 (python) S 29036 29038 28808 0 -1 4202560 36 932 0 0 0 0 0 0 18 0 35 0 548284740 756064256 19005 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699498929 0 0 16781312 2 18446604440278878976 0 0 -1 1 0 0 0
[pid=29038/tid=29094] ppid=29036 vsize=738344 CPUtime=0 cores=0-7
/proc/29038/task/29094/stat : 29094 (python) S 29036 29038 28808 0 -1 4202560 119 932 0 0 0 0 0 0 18 0 35 0 548285235 756064256 19005 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699498929 0 0 16781312 2 18446604440278878976 0 0 -1 6 0 0 0
[pid=29038/tid=29097] ppid=29036 vsize=738344 CPUtime=0 cores=0-7
/proc/29038/task/29097/stat : 29097 (python) S 29036 29038 28808 0 -1 4202560 33 932 0 0 0 0 0 0 18 0 35 0 548285240 756064256 19005 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699498929 0 0 16781312 2 18446604440278878976 0 0 -1 1 0 0 0
[pid=29038/tid=29101] ppid=29036 vsize=738344 CPUtime=0 cores=0-7
/proc/29038/task/29101/stat : 29101 (python) S 29036 29038 28808 0 -1 4202560 28 932 0 0 0 0 0 0 20 0 35 0 548285269 756064256 19005 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699498929 0 0 16781312 2 18446604440278878976 0 0 -1 4 0 0 0
[pid=29038/tid=29107] ppid=29036 vsize=738344 CPUtime=0 cores=0-7
/proc/29038/task/29107/stat : 29107 (python) S 29036 29038 28808 0 -1 4202560 39 932 0 0 0 0 0 0 18 0 35 0 548294771 756064256 19005 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699498929 0 0 16781312 2 18446604440278878976 0 0 -1 4 0 0 0
[pid=29038/tid=29110] ppid=29036 vsize=738344 CPUtime=0 cores=0-7
/proc/29038/task/29110/stat : 29110 (python) S 29036 29038 28808 0 -1 4202560 46 932 0 0 0 0 0 0 15 0 35 0 548294882 756064256 19005 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699498929 0 0 16781312 2 18446604440278878976 0 0 -1 4 0 0 0
[pid=29038/tid=29114] ppid=29036 vsize=738344 CPUtime=0 cores=0-7
/proc/29038/task/29114/stat : 29114 (python) S 29036 29038 28808 0 -1 4202560 35 932 0 0 0 0 0 0 18 0 35 0 548295304 756064256 19005 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699498929 0 0 16781312 2 18446604440278878976 0 0 -1 4 0 0 0
[pid=29038/tid=29126] ppid=29036 vsize=738344 CPUtime=0 cores=0-7
/proc/29038/task/29126/stat : 29126 (python) S 29036 29038 28808 0 -1 4202560 32 932 0 0 0 0 0 0 20 0 35 0 548314530 756064256 19005 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699498929 0 0 16781312 2 18446604440278878976 0 0 -1 1 0 0 0
[pid=29038/tid=29131] ppid=29036 vsize=738344 CPUtime=0 cores=0-7
/proc/29038/task/29131/stat : 29131 (python) S 29036 29038 28808 0 -1 4202560 30 932 0 0 0 0 0 0 19 0 35 0 548314856 756064256 19005 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699498929 0 0 16781312 2 18446604440278878976 0 0 -1 4 0 0 0
[pid=29038/tid=29134] ppid=29036 vsize=738344 CPUtime=0 cores=0-7
/proc/29038/task/29134/stat : 29134 (python) S 29036 29038 28808 0 -1 4202560 38 932 0 0 0 0 0 0 18 0 35 0 548315026 756064256 19005 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699498929 0 0 16781312 2 18446604440278878976 0 0 -1 4 0 0 0
[pid=29038/tid=29137] ppid=29036 vsize=738344 CPUtime=0 cores=0-7
/proc/29038/task/29137/stat : 29137 (python) S 29036 29038 28808 0 -1 4202560 28 932 0 0 0 0 0 0 22 0 35 0 548315036 756064256 19005 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699498929 0 0 16781312 2 18446604440278878976 0 0 -1 4 0 0 0
[pid=29038/tid=29143] ppid=29036 vsize=738344 CPUtime=0 cores=0-7
/proc/29038/task/29143/stat : 29143 (python) S 29036 29038 28808 0 -1 4202560 29 932 0 0 0 0 0 0 20 0 35 0 548324062 756064256 19005 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699498929 0 0 16781312 2 18446604440278878976 0 0 -1 1 0 0 0
[pid=29038/tid=29147] ppid=29036 vsize=738344 CPUtime=0 cores=0-7
/proc/29038/task/29147/stat : 29147 (python) S 29036 29038 28808 0 -1 4202560 156 932 0 0 0 0 0 0 20 0 35 0 548324563 756064256 19005 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699498929 0 0 16781312 2 18446604440278878976 0 0 -1 3 0 0 0
[pid=29038/tid=29151] ppid=29036 vsize=738344 CPUtime=0 cores=0-7
/proc/29038/task/29151/stat : 29151 (python) S 29036 29038 28808 0 -1 4202560 44 932 0 0 0 0 0 0 20 0 35 0 548324888 756064256 19005 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699498929 0 0 16781312 2 18446604440278878976 0 0 -1 4 0 0 0
[pid=29038/tid=29158] ppid=29036 vsize=738344 CPUtime=0 cores=0-7
/proc/29038/task/29158/stat : 29158 (python) S 29036 29038 28808 0 -1 4202560 39 932 0 0 0 0 0 0 20 0 35 0 548334002 756064256 19005 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699498929 0 0 16781312 2 18446604440278878976 0 0 -1 1 0 0 0
[pid=29038/tid=29161] ppid=29036 vsize=738344 CPUtime=0 cores=0-7
/proc/29038/task/29161/stat : 29161 (python) S 29036 29038 28808 0 -1 4202560 27 932 0 0 0 0 0 0 20 0 35 0 548334102 756064256 19005 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699498929 0 0 16781312 2 18446604440278878976 0 0 -1 4 0 0 0
[pid=29038/tid=29166] ppid=29036 vsize=738344 CPUtime=0 cores=0-7
/proc/29038/task/29166/stat : 29166 (python) S 29036 29038 28808 0 -1 4202560 25 932 0 0 0 0 0 0 20 0 35 0 548334598 756064256 19005 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699498929 0 0 16781312 2 18446604440278878976 0 0 -1 2 0 0 0
[pid=29038/tid=29169] ppid=29036 vsize=738344 CPUtime=0 cores=0-7
/proc/29038/task/29169/stat : 29169 (python) S 29036 29038 28808 0 -1 4202560 161 932 0 0 0 0 0 0 22 0 35 0 548334608 756064256 19005 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699498929 0 0 16781312 2 18446604440278878976 0 0 -1 2 0 0 0
[pid=29038/tid=29174] ppid=29036 vsize=738344 CPUtime=0 cores=0-7
/proc/29038/task/29174/stat : 29174 (python) S 29036 29038 28808 0 -1 4202560 39 932 0 0 0 0 0 0 20 0 35 0 548334921 756064256 19005 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699498929 0 0 16781312 2 18446604440278878976 0 0 -1 2 0 0 0
[pid=29038/tid=29180] ppid=29036 vsize=738344 CPUtime=0 cores=0-7
/proc/29038/task/29180/stat : 29180 (python) S 29036 29038 28808 0 -1 4202560 33 932 0 0 0 0 0 0 20 0 35 0 548344037 756064256 19005 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699498929 0 0 16781312 2 18446604440278878976 0 0 -1 2 0 0 0
[pid=29038/tid=29184] ppid=29036 vsize=738344 CPUtime=0 cores=0-7
/proc/29038/task/29184/stat : 29184 (python) S 29036 29038 28808 0 -1 4202560 36 932 0 0 0 0 0 0 20 0 35 0 548344232 756064256 19005 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699498929 0 0 16781312 2 18446604440278878976 0 0 -1 1 0 0 0
[pid=29038/tid=29188] ppid=29036 vsize=738344 CPUtime=0 cores=0-7
/proc/29038/task/29188/stat : 29188 (python) S 29036 29038 28808 0 -1 4202560 38 932 0 0 0 0 0 0 20 0 35 0 548344633 756064256 19005 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699498929 0 0 16781312 2 18446604440278878976 0 0 -1 2 0 0 0
[pid=29038/tid=29195] ppid=29036 vsize=738344 CPUtime=0 cores=0-7
/proc/29038/task/29195/stat : 29195 (python) S 29036 29038 28808 0 -1 4202560 36 932 0 0 0 0 0 0 20 0 35 0 548354074 756064256 19005 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699498929 0 0 16781312 2 18446604440278878976 0 0 -1 1 0 0 0
[pid=29038/tid=29199] ppid=29036 vsize=738344 CPUtime=0 cores=0-7
/proc/29038/task/29199/stat : 29199 (python) S 29036 29038 28808 0 -1 4202560 42 932 0 0 0 0 0 0 20 0 35 0 548354172 756064256 19005 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699498929 0 0 16781312 2 18446604440278878976 0 0 -1 2 0 0 0
[pid=29038/tid=29203] ppid=29036 vsize=738344 CPUtime=0 cores=0-7
/proc/29038/task/29203/stat : 29203 (python) S 29036 29038 28808 0 -1 4202560 29 932 0 0 0 0 0 0 20 0 35 0 548354269 756064256 19005 33554432000 4194304 5683644 140736016732768 18446744073709551615 212699498929 0 0 16781312 2 18446604440278878976 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 7883.1
Current children cumulated vsize (KiB) 32250572

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 29038 and gives
#  childrusage.ru_utime.tv_sec=2082
#  childrusage.ru_utime.tv_usec=339436
#  childrusage.ru_stime.tv_sec=9
#  childrusage.ru_stime.tv_usec=374574
# CPU time returned by wait4() is 2091.71
# while last known CPU time is 7883.1
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 998.916
CPU time (s): 7883.1
CPU user time (s): 7873.63
CPU system time (s): 9.47
CPU usage (%): 789.166
Max. virtual memory (cumulated for all children) (KiB): 32250572

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 2082.34
system time used= 9.37457
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 189547
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 12208
involuntary context switches= 12944

runsolver used 7.12392 second user time and 20.7598 second system time

The end

Launcher Data 

Begin job on node139 at 2011-05-02 22:02:47
IDJOB=3321743
IDBENCH=82521
IDSOLVER=1712
FILE ID=node139/3321743-1304366567
RUNJOBID= node139-1304366465-28822
PBS_JOBID= 13181734
Free space on /tmp= 73160 MiB

SOLVER NAME= borg-sat 11.03.02-random
BENCH NAME= SAT11/random/large/unif-k3-r4.2-v50000-c210000-S1969540674-065-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3321743-1304366567/watcher-3321743-1304366567 -o /tmp/evaluation-result-3321743-1304366567/solver-3321743-1304366567 -C 40000 -W 5100 -M 31000  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3321743-1304366567.cnf -seed 1719099220 -budget 40000 -cores 8

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= fcb14103324f734a2e69e02c062e67ed
RANDOM SEED=1719099220

node139.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.70
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5321.49
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      23868348 kB
Buffers:       1338064 kB
Cached:        4155696 kB
SwapCached:         60 kB
Active:        3280816 kB
Inactive:      5281720 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      23868348 kB
SwapTotal:    67111528 kB
SwapFree:     67111464 kB
Dirty:           36420 kB
Writeback:           0 kB
AnonPages:     3068512 kB
Mapped:          23988 kB
Slab:           435048 kB
PageTables:      19432 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 37403256 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73128 MiB
End job on node139 at 2011-05-02 22:19:28