Trace number 3385340

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-par-random (fixed)? (MO) 10948.2 1385.02

General information on the benchmark

Namerandom/large/
unif-k3-r4.2-v50000-c210000-S758437099-099-UNKNOWN.cnf
MD5SUM94d10ca78c6cee5b0badfb96b777ac08
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark838.323
Satisfiable
(Un)Satisfiability was proved
Number of variables49999
Number of clauses210000
Sum of the clauses size630000
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 3210000
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.31/0.47	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.31/0.47	c INFO: solving HOME/instance-3385340-1305565256.cnf
0.31/0.47	c INFO: computing task features
1.10/1.20	c INFO: computed variable-clause graph statistics
1.10/1.23	c INFO: computed clause balance statistics
4.70/4.86	c INFO: computed variable balance statistics
7.61/7.78	c INFO: computed small-clause counts
7.70/7.80	c INFO: computed Horn-clause counts
10.70/10.88	c INFO: computed variable graph statistics
10.70/10.88	c INFO: collected features for HOME/instance-3385340-1305565256.cnf in 10.38 s
10.70/10.91	c INFO: running hybridGM3@0 for 97 with 4989 remaining (b = 0.33)
10.80/10.94	c INFO: running hybridGM3@0 for 97 with 4989 remaining (b = 0.21)
10.80/10.97	c INFO: running hybridGM3@0 for 97 with 4989 remaining (b = 0.14)
10.80/11.00	c INFO: running hybridGM3@0 for 97 with 4989 remaining (b = 0.09)
10.91/11.03	c INFO: running hybridGM3@0 for 97 with 4989 remaining (b = 0.07)
10.91/11.07	c INFO: running hybridGM3@0 for 194 with 4989 remaining (b = 0.10)
11.00/11.10	c INFO: running hybridGM3@0 for 194 with 4989 remaining (b = 0.07)
11.00/11.13	c INFO: running hybridGM3@0 for 194 with 4989 remaining (b = 0.06)
811.62/111.30	c INFO: running hybridGM3@0 for 97 with 4889 remaining (b = 0.03)
811.62/111.33	c INFO: running hybridGM3@0 for 97 with 4889 remaining (b = 0.03)
811.62/111.37	c INFO: running hybridGM3@0 for 97 with 4889 remaining (b = 0.03)
812.33/111.40	c INFO: running hybridGM3@0 for 97 with 4889 remaining (b = 0.03)
812.33/111.46	c INFO: running hybridGM3@0 for 97 with 4889 remaining (b = 0.03)
1571.16/206.56	c INFO: running hybridGM3@100 for 97 with 4793 remaining (b = 0.03)
1571.16/206.59	c INFO: running hybridGM3@100 for 97 with 4793 remaining (b = 0.03)
1571.94/206.62	c INFO: running TNM@0 for 97 with 4793 remaining (b = 0.03)
1571.94/206.66	c INFO: running hybridGM3@100 for 97 with 4793 remaining (b = 0.02)
1580.66/207.74	c INFO: running hybridGM3@98 for 97 with 4792 remaining (b = 0.02)
1580.66/207.77	c INFO: running hybridGM3@0 for 97 with 4792 remaining (b = 0.02)
1620.49/212.73	c INFO: running hybridGM3@100 for 97 with 4787 remaining (b = 0.02)
1628.49/213.73	c INFO: running hybridGM3@95 for 97 with 4786 remaining (b = 0.02)
2371.43/306.92	c INFO: running hybridGM3@0 for 97 with 4693 remaining (b = 0.02)
2378.48/307.83	c INFO: running hybridGM3@0 for 194 with 4692 remaining (b = 0.04)
2378.48/307.87	c INFO: running TNM@0 for 97 with 4692 remaining (b = 0.02)
2379.29/307.93	c INFO: running hybridGM3@296 for 97 with 4692 remaining (b = 0.02)
2379.29/307.99	c INFO: running hybridGM3@193 for 97 with 4692 remaining (b = 0.02)
2380.10/308.03	c INFO: running hybridGM3@294 for 97 with 4692 remaining (b = 0.02)
2388.02/309.05	c INFO: running hybridGM3@99 for 97 with 4691 remaining (b = 0.02)
2395.96/310.01	c INFO: running hybridGM3@299 for 97 with 4690 remaining (b = 0.02)
3138.74/403.19	c INFO: running hybridGM3@391 for 97 with 4597 remaining (b = 0.01)
3139.50/403.29	c INFO: running TNM@0 for 97 with 4597 remaining (b = 0.02)
3146.66/404.19	c INFO: running hybridGM3@100 for 97 with 4596 remaining (b = 0.02)
3148.24/404.31	c INFO: running TNM@0 for 97 with 4596 remaining (b = 0.02)
3178.62/408.16	c INFO: running TNM@0 for 97 with 4592 remaining (b = 0.01)
3187.37/409.28	c INFO: running hybridGM3@293 for 97 with 4591 remaining (b = 0.02)
3203.21/411.29	c INFO: running TNM@0 for 97 with 4589 remaining (b = 0.01)
3890.32/497.49	c INFO: running hybridGM3@394 for 97 with 4503 remaining (b = 0.01)
3939.76/503.61	c INFO: running TNM@0 for 97 with 4496 remaining (b = 0.01)
3946.90/504.52	c INFO: running hybridGM3@394 for 97 with 4495 remaining (b = 0.01)
3946.90/504.56	c INFO: running TNM@0 for 97 with 4495 remaining (b = 0.01)
3947.58/504.62	c INFO: running TNM@94 for 1940 with 4495 remaining (b = 0.21)
3947.58/504.66	c INFO: running hybridGM3@101 for 97 with 4495 remaining (b = 0.02)
3977.87/508.47	c INFO: running hybridGM3@0 for 194 with 4492 remaining (b = 0.03)
3995.34/510.63	c INFO: running hybridGM3@392 for 97 with 4489 remaining (b = 0.01)
4665.51/594.79	c INFO: running TNM@0 for 97 with 4405 remaining (b = 0.01)
4706.93/599.95	c INFO: running hybridGM3@0 for 194 with 4400 remaining (b = 0.02)
4721.99/601.83	c INFO: running TNM@0 for 97 with 4398 remaining (b = 0.01)
4739.48/604.03	c INFO: running hybridGM3@0 for 194 with 4396 remaining (b = 0.02)
4745.10/604.89	c INFO: running adaptg2wsat2009++@0 for 194 with 4395 remaining (b = 0.01)
4770.41/607.94	c INFO: running TNM@0 for 97 with 4392 remaining (b = 0.01)
5480.33/697.19	c INFO: running TNM@0 for 97 with 4303 remaining (b = 0.01)
5520.13/702.17	c INFO: running TNM@0 for 2037 with 4298 remaining (b = 0.13)
5544.76/705.26	c INFO: running hybridGM3@98 for 97 with 4295 remaining (b = 0.01)
5568.61/708.29	c INFO: running hybridGM3@0 for 194 with 4292 remaining (b = 0.01)
6279.31/797.54	c INFO: running adaptg2wsat2009++@0 for 194 with 4202 remaining (b = 0.01)
6303.08/800.57	c INFO: running hybridGM3@0 for 194 with 4199 remaining (b = 0.01)
6306.21/800.92	c INFO: running hybridGM3@99 for 97 with 4199 remaining (b = 0.01)
6321.34/802.83	c INFO: running hybridGM3@94 for 97 with 4197 remaining (b = 0.01)
6350.77/806.60	c INFO: running adaptg2wsat2009++@97 for 97 with 4193 remaining (b = 0.01)
7064.49/896.23	c INFO: running hybridGM3@298 for 194 with 4104 remaining (b = 0.01)
7079.61/898.15	c INFO: running adaptg2wsat2009++@0 for 194 with 4102 remaining (b = 0.01)
7135.24/905.10	c INFO: running hybridGM3@99 for 97 with 4095 remaining (b = 0.01)
7173.41/909.95	c INFO: running hybridGM3@195 for 291 with 4090 remaining (b = 0.01)
7835.00/993.17	c INFO: running hybridGM3@195 for 291 with 4007 remaining (b = 0.01)
7869.18/997.41	c INFO: running hybridGM3@0 for 194 with 4003 remaining (b = 0.01)
7940.73/1006.44	c INFO: running FH@0 for 97 with 3994 remaining (b = 0.00)
8635.33/1093.84	c INFO: running adaptg2wsat2009++@192 for 97 with 3906 remaining (b = 0.00)
8635.33/1093.90	c INFO: running hybridGM3@393 for 97 with 3906 remaining (b = 0.01)
8737.78/1106.79	c INFO: running hybridGM3@196 for 291 with 3893 remaining (b = 0.01)
9393.53/1189.23	c INFO: running hybridGM3@194 for 291 with 3811 remaining (b = 0.01)
9408.62/1191.17	c INFO: running adaptg2wsat2009++@389 for 194 with 3809 remaining (b = 0.01)
9433.22/1194.24	c INFO: running TNM@0 for 97 with 3806 remaining (b = 0.00)
9476.94/1199.80	c INFO: running hybridGM3@390 for 97 with 3800 remaining (b = 0.01)
10139.00/1283.08	c INFO: running hybridGM3@390 for 97 with 3717 remaining (b = 0.01)
10230.40/1294.59	c INFO: running TNM@288 for 1843 with 3705 remaining (b = 0.08)
10235.13/1295.12	c INFO: running hybridGM3@100 for 97 with 3705 remaining (b = 0.00)
10931.46/1382.74	c INFO: running hybridGM3@201 for 291 with 3617 remaining (b = 0.01)
10944.89/1384.43	c INFO: running NCVWr@0 for 485 with 3616 remaining (b = 0.02)
10948.22/1384.93	c INFO: running hybridGM3@198 for 291 with 3615 remaining (b = 0.01)
10948.22/1384.93	Traceback (most recent call last):
10948.22/1384.93	  File "HOME/solve", line 33, in <module>
10948.22/1384.93	    plac.call(main)
10948.22/1384.93	  File "/home/evaluation/evaluation/tools64/lib/python2.7/site-packages/plac-0.8.1-py2.7.egg/plac_core.py", line 311, in call
10948.22/1384.93	    cmd, result = parser_from(obj).consume(arglist)
10948.22/1384.93	  File "/home/evaluation/evaluation/tools64/lib/python2.7/site-packages/plac-0.8.1-py2.7.egg/plac_core.py", line 192, in consume
10948.22/1384.93	    return cmd, self.func(*(args + varargs + extraopts), **kwargs)
10948.22/1384.93	  File "HOME/borg/build/lib/borg/tools/solve.py", line 84, in main
10948.22/1384.93	    (_, answer, _) = solver(input_path, budget / cores, cores)
10948.22/1384.93	  File "HOME/borg/build/lib/borg/bilevel.py", line 202, in __call__
10948.22/1384.93	    solver.go(max_cost * borg.defaults.machine_speed)
10948.22/1384.93	  File "HOME/borg/build/lib/borg/solvers.py", line 179, in go
10948.22/1384.93	    self._process.start()
10948.22/1384.93	  File "/home/evaluation/evaluation/tools64/lib/python2.7/multiprocessing/process.py", line 94, in start
10948.22/1384.93	    assert self._popen is None, 'cannot start a process twice'
10948.22/1384.93	AssertionError: cannot start a process twice

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3385340-1305565256/watcher-3385340-1305565256 -o /tmp/evaluation-result-3385340-1305565256/solver-3385340-1305565256 -C 40000 -W 5100 -M 31000 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3385340-1305565256.cnf -seed 6825901 -budget 40000 -cores 8 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.97 4.98 2.14 2/325 4149
/proc/meminfo: memFree=18702068/32951124 swapFree=67056788/67111528
[pid=4149] ppid=4147 vsize=15380 CPUtime=0 cores=0-7
/proc/4149/stat : 4149 (python) S 4147 4149 3941 0 -1 4202496 793 0 0 0 0 0 0 0 20 0 1 0 668123550 15749120 681 33554432000 4194304 5683644 140733438452224 18446744073709551615 271550535680 0 2147483391 16781312 2 18446744071699469502 0 0 17 6 0 0 0
/proc/4149/statm: 3845 681 322 364 0 332 0

[startup+0.0360919 s]
/proc/loadavg: 7.97 4.98 2.14 2/325 4149
/proc/meminfo: memFree=18702068/32951124 swapFree=67056788/67111528
[pid=4149] ppid=4147 vsize=18880 CPUtime=0.02 cores=0-7
/proc/4149/stat : 4149 (python) R 4147 4149 3941 0 -1 4202496 1283 0 0 0 2 0 0 0 20 0 1 0 668123550 19333120 1137 33554432000 4194304 5683644 140733438452224 18446744073709551615 4509260 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/4149/statm: 4720 1137 421 364 0 689 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 18880

[startup+0.101069 s]
/proc/loadavg: 7.97 4.98 2.14 2/325 4149
/proc/meminfo: memFree=18702068/32951124 swapFree=67056788/67111528
[pid=4149] ppid=4147 vsize=104224 CPUtime=0.07 cores=0-7
/proc/4149/stat : 4149 (python) S 4147 4149 3941 0 -1 4202496 3069 0 0 0 5 2 0 0 17 0 1 0 668123550 106725376 2751 33554432000 4194304 5683644 140733438452224 18446744073709551615 271550535680 0 2147483391 16781312 2 18446744071699469502 0 0 17 6 0 0 0
/proc/4149/statm: 26056 2751 847 364 0 1804 0
Current children cumulated CPU time (s) 0.07
Current children cumulated vsize (KiB) 104224

[startup+0.301035 s]
/proc/loadavg: 7.97 4.98 2.14 2/325 4149
/proc/meminfo: memFree=18702068/32951124 swapFree=67056788/67111528
[pid=4149] ppid=4147 vsize=170748 CPUtime=0.23 cores=0-7
/proc/4149/stat : 4149 (python) R 4147 4149 3941 0 -1 4202496 6730 936 1 0 17 6 0 0 17 0 1 0 668123550 174845952 4805 33554432000 4194304 5683644 140733438452224 18446744073709551615 271550535680 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/4149/statm: 42687 4805 1336 364 0 3452 0
Current children cumulated CPU time (s) 0.23
Current children cumulated vsize (KiB) 170748

[startup+0.700957 s]
/proc/loadavg: 7.97 4.98 2.14 2/325 4149
/proc/meminfo: memFree=18702068/32951124 swapFree=67056788/67111528
[pid=4149] ppid=4147 vsize=296912 CPUtime=0.6 cores=0-7
/proc/4149/stat : 4149 (python) R 4147 4149 3941 0 -1 4202496 13814 936 1 0 51 9 0 0 17 0 1 0 668123550 304037888 11502 33554432000 4194304 5683644 140733438452224 18446744073709551615 4602399 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/4149/statm: 74228 11503 1761 364 0 10152 0
Current children cumulated CPU time (s) 0.6
Current children cumulated vsize (KiB) 296912

[startup+1.50081 s]
/proc/loadavg: 7.97 4.98 2.14 2/326 4154
/proc/meminfo: memFree=18644524/32951124 swapFree=67056788/67111528
[pid=4149] ppid=4147 vsize=327992 CPUtime=1.4 cores=0-7
/proc/4149/stat : 4149 (python) R 4147 4149 3941 0 -1 4202496 27446 936 1 0 129 11 0 0 19 0 1 0 668123550 335863808 19383 33554432000 4194304 5683644 140733438452224 18446744073709551615 271550204805 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/4149/statm: 81998 19383 1826 364 0 17922 0
Current children cumulated CPU time (s) 1.4
Current children cumulated vsize (KiB) 327992

[startup+3.10052 s]
/proc/loadavg: 7.97 4.98 2.14 2/326 4154
/proc/meminfo: memFree=18630760/32951124 swapFree=67056788/67111528
[pid=4149] ppid=4147 vsize=327992 CPUtime=3 cores=0-7
/proc/4149/stat : 4149 (python) R 4147 4149 3941 0 -1 4202496 27446 936 1 0 289 11 0 0 22 0 1 0 668123550 335863808 19383 33554432000 4194304 5683644 140733438452224 18446744073709551615 47352179024635 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/4149/statm: 81998 19383 1826 364 0 17922 0
Current children cumulated CPU time (s) 3
Current children cumulated vsize (KiB) 327992

[startup+6.30094 s]
/proc/loadavg: 7.41 4.91 2.13 2/326 4163
/proc/meminfo: memFree=18630512/32951124 swapFree=67056788/67111528
[pid=4149] ppid=4147 vsize=327992 CPUtime=6.2 cores=0-7
/proc/4149/stat : 4149 (python) R 4147 4149 3941 0 -1 4202496 27447 936 1 0 609 11 0 0 25 0 1 0 668123550 335863808 19384 33554432000 4194304 5683644 140733438452224 18446744073709551615 271550234057 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/4149/statm: 81998 19384 1827 364 0 17922 0
Current children cumulated CPU time (s) 6.2
Current children cumulated vsize (KiB) 327992

[startup+12.7017 s]
/proc/loadavg: 6.90 4.85 2.13 9/350 4187
/proc/meminfo: memFree=18471508/32951124 swapFree=67056788/67111528
[pid=4149] ppid=4147 vsize=471684 CPUtime=11.03 cores=0-7
/proc/4149/stat : 4149 (python) S 4147 4149 3941 0 -1 4202496 33279 936 1 0 1089 14 0 0 21 0 9 0 668123550 483004416 18667 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562364363 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/4149/statm: 117921 18667 1889 364 0 53818 0
[pid=4164] ppid=4149 vsize=324112 CPUtime=0 cores=0-7
/proc/4164/stat : 4164 (python) S 4149 4149 3941 0 -1 4202560 738 0 0 0 0 0 0 0 18 0 1 0 668124641 331890688 16934 33554432000 4194304 5683644 140733438452224 18446744073709551615 271550557711 0 0 16781312 514 1278865068305416191 0 0 17 1 0 0 0
/proc/4164/statm: 81028 16934 353 364 0 16946 0
[pid=4166] ppid=4164 vsize=276236 CPUtime=1.77 cores=0-7
/proc/4166/stat : 4166 (hybridGM3) R 4164 4166 4166 0 -1 4202496 5078 0 0 0 177 0 0 0 25 0 1 0 668124642 282865664 4815 33554432000 134512640 135204385 4291835760 18446744073709551615 134543975 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/4166/statm: 69059 4815 57 169 0 68889 0
[pid=4167] ppid=4149 vsize=334368 CPUtime=0 cores=0-7
/proc/4167/stat : 4167 (python) S 4149 4149 3941 0 -1 4202560 742 0 0 0 0 0 0 0 18 0 1 0 668124645 342392832 16932 33554432000 4194304 5683644 140733438452224 18446744073709551615 271550557766 0 0 16781312 514 6320459308559499263 0 0 17 1 0 0 0
/proc/4167/statm: 83592 16932 347 364 0 19507 0
[pid=4169] ppid=4167 vsize=276240 CPUtime=1.74 cores=0-7
/proc/4169/stat : 4169 (hybridGM3) R 4167 4169 4169 0 -1 4202496 5059 0 0 0 174 0 0 0 25 0 1 0 668124645 282869760 4796 33554432000 134512640 135204385 4289930992 18446744073709551615 134543968 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/4169/statm: 69060 4796 57 169 0 68890 0
[pid=4170] ppid=4149 vsize=410160 CPUtime=0 cores=0-7
/proc/4170/stat : 4170 (python) S 4149 4149 3941 0 -1 4202560 701 0 0 0 0 0 0 0 18 0 1 0 668124648 420003840 17110 33554432000 4194304 5683644 140733438452224 18446744073709551615 271550557766 0 0 16781312 514 6320463156850196479 0 0 17 1 0 0 0
/proc/4170/statm: 102540 17110 348 364 0 38452 0
[pid=4172] ppid=4170 vsize=276236 CPUtime=1.71 cores=0-7
/proc/4172/stat : 4172 (hybridGM3) R 4170 4172 4172 0 -1 4202496 5025 0 0 0 171 0 0 0 25 0 1 0 668124648 282865664 4774 33554432000 134512640 135204385 4289067600 18446744073709551615 134543975 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/4172/statm: 69059 4774 57 169 0 68889 0
[pid=4173] ppid=4149 vsize=420416 CPUtime=0 cores=0-7
/proc/4173/stat : 4173 (python) S 4149 4149 3941 0 -1 4202560 713 0 0 0 0 0 0 0 18 0 1 0 668124651 430505984 17071 33554432000 4194304 5683644 140733438452224 18446744073709551615 271550557766 0 0 16781312 514 16431982203715452927 0 0 17 1 0 0 0
/proc/4173/statm: 105104 17071 348 364 0 41013 0
[pid=4175] ppid=4173 vsize=276232 CPUtime=1.68 cores=0-7
/proc/4175/stat : 4175 (hybridGM3) R 4173 4175 4175 0 -1 4202496 5006 0 0 0 168 0 0 0 25 0 1 0 668124651 282861568 4754 33554432000 134512640 135204385 4290105584 18446744073709551615 134543968 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/4175/statm: 69058 4754 57 169 0 68888 0
[pid=4176] ppid=4149 vsize=430672 CPUtime=0 cores=0-7
/proc/4176/stat : 4176 (python) S 4149 4149 3941 0 -1 4202560 722 0 0 0 0 0 0 0 18 0 1 0 668124654 441008128 17093 33554432000 4194304 5683644 140733438452224 18446744073709551615 271550557766 0 0 16781312 514 4039860261685821439 0 0 17 1 0 0 0
/proc/4176/statm: 107668 17093 348 364 0 43574 0
[pid=4178] ppid=4176 vsize=276236 CPUtime=1.63 cores=0-7
/proc/4178/stat : 4178 (hybridGM3) R 4176 4178 4178 0 -1 4202496 4962 0 0 0 163 0 0 0 25 0 1 0 668124655 282865664 4707 33554432000 134512640 135204385 4289797328 18446744073709551615 134543975 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/4178/statm: 69059 4708 57 169 0 68889 0
[pid=4179] ppid=4149 vsize=440928 CPUtime=0 cores=0-7
/proc/4179/stat : 4179 (python) S 4149 4149 3941 0 -1 4202560 728 0 0 0 0 0 0 0 18 0 1 0 668124657 451510272 17114 33554432000 4194304 5683644 140733438452224 18446744073709551615 271550557766 0 0 16781312 514 4138250334575263743 0 0 17 1 0 0 0
/proc/4179/statm: 110232 17114 348 364 0 46135 0
[pid=4181] ppid=4179 vsize=276232 CPUtime=1.6 cores=0-7
/proc/4181/stat : 4181 (hybridGM3) R 4179 4181 4181 0 -1 4202496 4952 0 0 0 160 0 0 0 25 0 1 0 668124658 282861568 4694 33554432000 134512640 135204385 4294360432 18446744073709551615 134543975 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/4181/statm: 69058 4694 57 169 0 68888 0
[pid=4182] ppid=4149 vsize=451184 CPUtime=0 cores=0-7
/proc/4182/stat : 4182 (python) S 4149 4149 3941 0 -1 4202560 738 0 0 0 0 0 0 0 18 0 1 0 668124661 462012416 17135 33554432000 4194304 5683644 140733438452224 18446744073709551615 271550557766 0 0 16781312 514 11969697000636547071 0 0 17 1 0 0 0
/proc/4182/statm: 112796 17135 348 364 0 48696 0
[pid=4184] ppid=4182 vsize=276236 CPUtime=1.55 cores=0-7
/proc/4184/stat : 4184 (hybridGM3) R 4182 4184 4184 0 -1 4202496 4926 0 0 0 155 0 0 0 25 0 1 0 668124663 282865664 4666 33554432000 134512640 135204385 4291769376 18446744073709551615 134543975 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/4184/statm: 69059 4666 57 169 0 68889 0
[pid=4185] ppid=4149 vsize=461440 CPUtime=0 cores=0-7
/proc/4185/stat : 4185 (python) S 4149 4149 3941 0 -1 4202560 776 0 0 0 0 0 0 0 18 0 1 0 668124664 472514560 17128 33554432000 4194304 5683644 140733438452224 18446744073709551615 271550557766 0 0 16781312 514 1278868916596113407 0 0 17 1 0 0 0
/proc/4185/statm: 115360 17128 354 364 0 51257 0
[pid=4187] ppid=4185 vsize=276236 CPUtime=1.53 cores=0-7
/proc/4187/stat : 4187 (hybridGM3) R 4185 4187 4187 0 -1 4202496 4912 0 0 0 153 0 0 0 25 0 1 0 668124666 282865664 4647 33554432000 134512640 135204385 4290616720 18446744073709551615 134543993 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/4187/statm: 69059 4647 57 169 0 68889 0
[pid=4149/tid=4165] ppid=4147 vsize=471684 CPUtime=0 cores=0-7
/proc/4149/task/4165/stat : 4165 (python) S 4147 4149 3941 0 -1 4202560 25 936 0 0 0 0 0 0 25 0 9 0 668124642 483004416 18667 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562361265 0 0 16781312 2 18446604451404639104 0 0 -1 0 0 0 0
[pid=4149/tid=4168] ppid=4147 vsize=471684 CPUtime=0 cores=0-7
/proc/4149/task/4168/stat : 4168 (python) S 4147 4149 3941 0 -1 4202560 22 936 0 0 0 0 0 0 18 0 9 0 668124645 483004416 18667 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562361265 0 0 16781312 2 18446604451404639104 0 0 -1 3 0 0 0
[pid=4149/tid=4171] ppid=4147 vsize=471684 CPUtime=0 cores=0-7
/proc/4149/task/4171/stat : 4171 (python) S 4147 4149 3941 0 -1 4202560 85 936 0 0 0 0 0 0 19 0 9 0 668124648 483004416 18667 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562361265 0 0 16781312 2 18446604451404639104 0 0 -1 0 0 0 0
[pid=4149/tid=4174] ppid=4147 vsize=471684 CPUtime=0 cores=0-7
/proc/4149/task/4174/stat : 4174 (python) S 4147 4149 3941 0 -1 4202560 31 936 0 0 0 0 0 0 18 0 9 0 668124651 483004416 18667 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562361265 0 0 16781312 2 18446604451404639104 0 0 -1 5 0 0 0
[pid=4149/tid=4177] ppid=4147 vsize=471684 CPUtime=0 cores=0-7
/proc/4149/task/4177/stat : 4177 (python) S 4147 4149 3941 0 -1 4202560 33 936 0 0 0 0 0 0 19 0 9 0 668124654 483004416 18667 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562361265 0 0 16781312 2 18446604451404639104 0 0 -1 0 0 0 0
[pid=4149/tid=4180] ppid=4147 vsize=471684 CPUtime=0 cores=0-7
/proc/4149/task/4180/stat : 4180 (python) S 4147 4149 3941 0 -1 4202560 32 936 0 0 0 0 0 0 20 0 9 0 668124658 483004416 18667 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562361265 0 0 16781312 2 18446604451404639104 0 0 -1 7 0 0 0
[pid=4149/tid=4183] ppid=4147 vsize=471684 CPUtime=0 cores=0-7
/proc/4149/task/4183/stat : 4183 (python) S 4147 4149 3941 0 -1 4202560 33 936 0 0 0 0 0 0 21 0 9 0 668124661 483004416 18667 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562361265 0 0 16781312 2 18446604451404639104 0 0 -1 1 0 0 0
[pid=4149/tid=4186] ppid=4147 vsize=471684 CPUtime=0 cores=0-7
/proc/4149/task/4186/stat : 4186 (python) S 4147 4149 3941 0 -1 4202560 31 936 0 0 0 0 0 0 22 0 9 0 668124664 483004416 18667 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562361265 0 0 16781312 2 18446604451404639104 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 24.24
Current children cumulated vsize (KiB) 5954848

[startup+25.5014 s]
/proc/loadavg: 7.14 5.00 2.22 9/350 4188
/proc/meminfo: memFree=18310860/32951124 swapFree=67056788/67111528
[pid=4149] ppid=4147 vsize=471684 CPUtime=11.03 cores=0-7
/proc/4149/stat : 4149 (python) S 4147 4149 3941 0 -1 4202496 33279 936 1 0 1089 14 0 0 21 0 9 0 668123550 483004416 18667 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562364363 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/4149/statm: 117921 18667 1889 364 0 53818 0
[pid=4164] ppid=4149 vsize=324112 CPUtime=0.03 cores=0-7
/proc/4164/stat : 4164 (python) S 4149 4149 3941 0 -1 4202560 909 0 0 0 1 2 0 0 15 0 1 0 668124641 331890688 16957 33554432000 4194304 5683644 140733438452224 18446744073709551615 271550557711 0 0 16781312 514 487725765895356676 0 0 17 1 0 0 0
/proc/4164/statm: 81028 16957 376 364 0 16946 0
[pid=4166] ppid=4164 vsize=276240 CPUtime=14.58 cores=0-7
/proc/4166/stat : 4166 (hybridGM3) R 4164 4166 4166 0 -1 4202496 9510 0 0 0 1457 1 0 0 25 0 1 0 668124642 282869760 9247 33554432000 134512640 135204385 4291835760 18446744073709551615 134536744 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/4166/statm: 69060 9247 64 169 0 68890 0
[pid=4167] ppid=4149 vsize=334368 CPUtime=0.03 cores=0-7
/proc/4167/stat : 4167 (python) S 4149 4149 3941 0 -1 4202560 918 0 0 0 1 2 0 0 15 0 1 0 668124645 342392832 16955 33554432000 4194304 5683644 140733438452224 18446744073709551615 271550557766 0 0 16781312 514 487725765895356676 0 0 17 1 0 0 0
/proc/4167/statm: 83592 16955 370 364 0 19507 0
[pid=4169] ppid=4167 vsize=276244 CPUtime=14.55 cores=0-7
/proc/4169/stat : 4169 (hybridGM3) R 4167 4169 4169 0 -1 4202496 9519 0 0 0 1454 1 0 0 25 0 1 0 668124645 282873856 9256 33554432000 134512640 135204385 4289930992 18446744073709551615 134545864 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/4169/statm: 69061 9256 64 169 0 68891 0
[pid=4170] ppid=4149 vsize=410160 CPUtime=0.02 cores=0-7
/proc/4170/stat : 4170 (python) S 4149 4149 3941 0 -1 4202560 780 0 0 0 1 1 0 0 15 0 1 0 668124648 420003840 17133 33554432000 4194304 5683644 140733438452224 18446744073709551615 271550557766 0 0 16781312 514 487725765895356673 0 0 17 1 0 0 0
/proc/4170/statm: 102540 17133 371 364 0 38452 0
[pid=4172] ppid=4170 vsize=276240 CPUtime=14.51 cores=0-7
/proc/4172/stat : 4172 (hybridGM3) R 4170 4172 4172 0 -1 4202496 9503 0 0 0 1450 1 0 0 25 0 1 0 668124648 282869760 9252 33554432000 134512640 135204385 4289067600 18446744073709551615 134536744 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/4172/statm: 69060 9252 64 169 0 68890 0
[pid=4173] ppid=4149 vsize=420416 CPUtime=0.03 cores=0-7
/proc/4173/stat : 4173 (python) S 4149 4149 3941 0 -1 4202560 794 0 0 0 1 2 0 0 15 0 1 0 668124651 430505984 17094 33554432000 4194304 5683644 140733438452224 18446744073709551615 271550557766 0 0 16781312 514 487725765895356676 0 0 17 1 0 0 0
/proc/4173/statm: 105104 17094 371 364 0 41013 0
[pid=4175] ppid=4173 vsize=276236 CPUtime=14.48 cores=0-7
/proc/4175/stat : 4175 (hybridGM3) R 4173 4175 4175 0 -1 4202496 9502 0 0 0 1447 1 0 0 25 0 1 0 668124651 282865664 9250 33554432000 134512640 135204385 4290105584 18446744073709551615 134537403 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/4175/statm: 69059 9250 64 169 0 68889 0
[pid=4176] ppid=4149 vsize=430672 CPUtime=0.02 cores=0-7
/proc/4176/stat : 4176 (python) S 4149 4149 3941 0 -1 4202560 800 0 0 0 1 1 0 0 15 0 1 0 668124654 441008128 17116 33554432000 4194304 5683644 140733438452224 18446744073709551615 271550557766 0 0 16781312 514 487725765895356676 0 0 17 1 0 0 0
/proc/4176/statm: 107668 17116 371 364 0 43574 0
[pid=4178] ppid=4176 vsize=276240 CPUtime=14.19 cores=0-7
/proc/4178/stat : 4178 (hybridGM3) R 4176 4178 4178 0 -1 4202496 9501 0 0 0 1418 1 0 0 25 0 1 0 668124655 282869760 9246 33554432000 134512640 135204385 4289797328 18446744073709551615 134537403 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/4178/statm: 69060 9246 64 169 0 68890 0

################
# More data... #
################

[pid=4271] ppid=4269 vsize=471116 CPUtime=100.21 cores=0-7
/proc/4271/stat : 4271 (TNM) T 4269 4271 4271 0 -1 4202496 8451 0 0 0 10020 1 0 0 25 0 1 0 668184346 482422784 8171 33554432000 134512640 135008748 4290238816 18446744073709551615 134528974 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/4271/statm: 117779 8171 46 122 0 116679 0
[pid=4269/tid=4284] ppid=4149 vsize=625536 CPUtime=0 cores=0-7
/proc/4269/task/4284/stat : 4284 (python) S 4149 4149 3941 0 -1 4202560 90 0 0 0 0 0 0 0 19 0 2 0 668194374 640548864 17526 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562361265 0 0 16781312 514 18446604440773170240 0 0 -1 6 0 0 0
[pid=4275] ppid=4149 vsize=635792 CPUtime=0.29 cores=0-7
/proc/4275/stat : 4275 (python) S 4149 4149 3941 0 -1 4202560 1178 0 0 0 13 16 0 0 19 0 2 0 668193270 651051008 17521 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562364363 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/4275/statm: 158948 17521 400 364 0 94794 0
[pid=4277] ppid=4275 vsize=471116 CPUtime=99.64 cores=0-7
/proc/4277/stat : 4277 (TNM) T 4275 4277 4277 0 -1 4202496 8454 0 0 0 9962 2 0 0 25 0 1 0 668193271 482422784 8172 33554432000 134512640 135008748 4291434832 18446744073709551615 134524772 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/4277/statm: 117779 8172 46 122 0 116679 0
[pid=4275/tid=4290] ppid=4149 vsize=635792 CPUtime=0 cores=0-7
/proc/4275/task/4290/stat : 4290 (python) S 4149 4149 3941 0 -1 4202560 87 0 0 0 0 0 0 0 19 0 2 0 668203299 651051008 17521 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562361265 0 0 16781312 514 18446604440773169344 0 0 -1 0 0 0 0
[pid=4280] ppid=4149 vsize=646048 CPUtime=1.93 cores=0-7
/proc/4280/stat : 4280 (python) S 4149 4149 3941 0 -1 4202560 985 0 0 0 99 94 0 0 18 0 1 0 668193768 661553152 17514 33554432000 4194304 5683644 140733438452224 18446744073709551615 271550557766 0 0 16781312 514 487725765895356673 0 0 17 4 0 0 0
/proc/4280/statm: 161512 17514 372 364 0 97355 0
[pid=4282] ppid=4280 vsize=471116 CPUtime=659.66 cores=0-7
/proc/4282/stat : 4282 (TNM) R 4280 4282 4282 0 -1 4202496 8453 0 0 0 65960 6 0 0 25 0 1 0 668193769 482422784 8172 33554432000 134512640 135008748 4290462864 18446744073709551615 134527320 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/4282/statm: 117779 8172 46 122 0 116679 0
[pid=4285] ppid=4149 vsize=656304 CPUtime=0.84 cores=0-7
/proc/4285/stat : 4285 (python) S 4149 4149 3941 0 -1 4202560 1208 0 0 0 47 37 0 0 15 0 2 0 668194380 672055296 17592 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562364363 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/4285/statm: 164076 17592 400 364 0 99916 0
[pid=4287] ppid=4285 vsize=276240 CPUtime=296.76 cores=0-7
/proc/4287/stat : 4287 (hybridGM3) T 4285 4287 4287 0 -1 4202496 9534 0 0 0 29672 4 0 0 25 0 1 0 668194380 282869760 9255 33554432000 134512640 135204385 4292841536 18446744073709551615 134536663 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/4287/statm: 69060 9255 64 169 0 68890 0
[pid=4285/tid=4306] ppid=4149 vsize=656304 CPUtime=0 cores=0-7
/proc/4285/task/4306/stat : 4306 (python) S 4149 4149 3941 0 -1 4202560 92 0 0 0 0 0 0 0 15 0 2 0 668214055 672055296 17592 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562361265 0 0 16781312 514 18446604453864053696 0 0 -1 4 0 0 0
[pid=4291] ppid=4149 vsize=666560 CPUtime=0.56 cores=0-7
/proc/4291/stat : 4291 (python) S 4149 4149 3941 0 -1 4202560 1205 0 0 0 27 29 0 0 19 0 2 0 668203305 682557440 17551 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562364363 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/4291/statm: 166640 17551 401 364 0 102477 0
[pid=4293] ppid=4291 vsize=438604 CPUtime=194.72 cores=0-7
/proc/4293/stat : 4293 (adaptg2wsat2009) T 4291 4293 4293 0 -1 4202496 8784 0 0 0 19470 2 0 0 25 0 1 0 668203306 449130496 8498 33554432000 134512640 135107411 4287288272 18446744073709551615 134532135 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/4293/statm: 109651 8498 59 146 0 108527 0
[pid=4291/tid=4309] ppid=4149 vsize=666560 CPUtime=0 cores=0-7
/proc/4291/task/4309/stat : 4309 (python) S 4149 4149 3941 0 -1 4202560 92 0 0 0 0 0 0 0 19 0 2 0 668222862 682557440 17551 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562361265 0 0 16781312 514 18446604451404637312 0 0 -1 3 0 0 0
[pid=4295] ppid=4149 vsize=676816 CPUtime=0.56 cores=0-7
/proc/4295/stat : 4295 (python) S 4149 4149 3941 0 -1 4202560 1234 0 0 0 26 30 0 0 16 0 2 0 668203607 693059584 17616 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562364363 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/4295/statm: 169204 17616 406 364 0 105038 0
[pid=4297] ppid=4295 vsize=276236 CPUtime=195.47 cores=0-7
/proc/4297/stat : 4297 (hybridGM3) T 4295 4297 4297 0 -1 4202496 9543 0 0 0 19545 2 0 0 25 0 1 0 668203608 282865664 9254 33554432000 134512640 135204385 4287880080 18446744073709551615 134536605 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/4297/statm: 69059 9254 64 169 0 68889 0
[pid=4295/tid=4310] ppid=4149 vsize=676816 CPUtime=0 cores=0-7
/proc/4295/task/4310/stat : 4310 (python) S 4149 4149 3941 0 -1 4202560 93 0 0 0 0 0 0 0 16 0 2 0 668223285 693059584 17616 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562361265 0 0 16781312 514 18446604452867954816 0 0 -1 1 0 0 0
[pid=4303] ppid=4149 vsize=687072 CPUtime=0.57 cores=0-7
/proc/4303/stat : 4303 (python) S 4149 4149 3941 0 -1 4202560 1246 0 0 0 27 30 0 0 19 0 2 0 668213366 703561728 17578 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562364363 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/4303/statm: 171768 17578 407 364 0 107599 0
[pid=4305] ppid=4303 vsize=438604 CPUtime=195.5 cores=0-7
/proc/4305/stat : 4305 (adaptg2wsat2009) T 4303 4305 4305 0 -1 4202496 8785 0 0 0 19548 2 0 0 25 0 1 0 668213367 449130496 8498 33554432000 134512640 135107411 4294432560 18446744073709551615 134531435 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/4305/statm: 109651 8498 59 146 0 108527 0
[pid=4303/tid=4320] ppid=4149 vsize=687072 CPUtime=0 cores=0-7
/proc/4303/task/4320/stat : 4320 (python) S 4149 4149 3941 0 -1 4202560 96 0 0 0 0 0 0 0 19 0 2 0 668232928 703561728 17578 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562361265 0 0 16781312 514 18446604445520115776 0 0 -1 0 0 0 0
[pid=4311] ppid=4149 vsize=697328 CPUtime=0.82 cores=0-7
/proc/4311/stat : 4311 (python) S 4149 4149 3941 0 -1 4202560 1260 0 0 0 43 39 0 0 15 0 2 0 668223292 714063872 17526 33554432000 4194304 5683644 140733438452224 18446744073709551615 271550557766 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/4311/statm: 174332 17526 406 364 0 110160 0
[pid=4313] ppid=4311 vsize=276236 CPUtime=283.98 cores=0-7
/proc/4313/stat : 4313 (hybridGM3) R 4311 4313 4313 0 -1 4202496 9545 0 0 0 28396 2 0 0 25 0 1 0 668223293 282865664 9255 33554432000 134512640 135204385 4293870112 18446744073709551615 134536843 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/4313/statm: 69059 9255 64 169 0 68889 0
[pid=4311/tid=4325] ppid=4149 vsize=697328 CPUtime=0 cores=0-7
/proc/4311/task/4325/stat : 4325 (python) S 4149 4149 3941 0 -1 4202560 97 0 0 0 0 0 0 0 16 0 2 0 668242968 714063872 17526 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562361265 0 0 16781312 514 18446604441631942848 0 0 -1 4 0 0 0
[pid=4315] ppid=4149 vsize=707584 CPUtime=0.29 cores=0-7
/proc/4315/stat : 4315 (python) S 4149 4149 3941 0 -1 4202560 1256 0 0 0 16 13 0 0 18 0 2 0 668224195 724566016 17623 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562364363 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/4315/statm: 176896 17623 407 364 0 112721 0
[pid=4317] ppid=4315 vsize=266472 CPUtime=100.24 cores=0-7
/proc/4317/stat : 4317 (FH) T 4315 4317 4317 0 -1 4202496 8673 0 0 0 10023 1 0 0 25 0 1 0 668224196 272867328 8381 33554432000 134512640 134936988 4288383712 18446744073709551615 134553267 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/4317/statm: 66618 8381 41 104 0 66122 0
[pid=4315/tid=4321] ppid=4149 vsize=707584 CPUtime=0 cores=0-7
/proc/4315/task/4321/stat : 4321 (python) S 4149 4149 3941 0 -1 4202560 90 0 0 0 0 0 0 0 19 0 2 0 668234224 724566016 17623 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562361265 0 0 16781312 514 18446604441631941056 0 0 -1 3 0 0 0
[pid=4326] ppid=4149 vsize=717840 CPUtime=0.55 cores=0-7
/proc/4326/stat : 4326 (python) S 4149 4149 3941 0 -1 4202560 1281 0 0 0 28 27 0 0 18 0 2 0 668242975 735068160 17577 33554432000 4194304 5683644 140733438452224 18446744073709551615 271550557766 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/4326/statm: 179460 17577 406 364 0 115282 0
[pid=4328] ppid=4326 vsize=471116 CPUtime=188.85 cores=0-7
/proc/4328/stat : 4328 (TNM) R 4326 4328 4328 0 -1 4202496 8463 0 0 0 18884 1 0 0 25 0 1 0 668242975 482422784 8172 33554432000 134512640 135008748 4291333120 18446744073709551615 134529195 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/4328/statm: 117779 8172 46 122 0 116679 0
[pid=4326/tid=4331] ppid=4149 vsize=717840 CPUtime=0 cores=0-7
/proc/4326/task/4331/stat : 4331 (python) S 4149 4149 3941 0 -1 4202560 99 0 0 0 0 0 0 0 19 0 2 0 668253004 735068160 17577 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562361265 0 0 16781312 514 18446604443723963392 0 0 -1 7 0 0 0
[pid=4335] ppid=4149 vsize=728096 CPUtime=0 cores=0-7
/proc/4335/stat : 4335 (python) S 4149 4149 3941 0 -1 4202560 921 0 0 0 0 0 0 0 19 0 1 0 668261994 745570304 17587 33554432000 4194304 5683644 140733438452224 18446744073709551615 271550557766 0 0 16781312 514 4825053675795775487 0 0 17 6 0 0 0
/proc/4335/statm: 182024 17587 346 364 0 117843 0
[pid=4337] ppid=4335 vsize=426872 CPUtime=0.37 cores=0-7
/proc/4337/stat : 4337 (NCVWr) R 4335 4337 4337 0 -1 4202496 5166 0 0 0 36 1 0 0 21 0 1 0 668261995 437116928 4879 33554432000 134512640 135113558 4289992096 18446744073709551615 134526301 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/4337/statm: 106718 4879 55 147 0 105593 0
[pid=4149/tid=4165] ppid=4147 vsize=738340 CPUtime=0 cores=0-7
/proc/4149/task/4165/stat : 4165 (python) S 4147 4149 3941 0 -1 4202560 40 936 0 0 0 0 0 0 18 0 35 0 668124642 756060160 19187 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562361265 0 0 16781312 2 18446604451404639104 0 0 -1 0 0 0 0
[pid=4149/tid=4168] ppid=4147 vsize=738340 CPUtime=0 cores=0-7
/proc/4149/task/4168/stat : 4168 (python) S 4147 4149 3941 0 -1 4202560 38 936 0 0 0 0 0 0 15 0 35 0 668124645 756060160 19187 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562361265 0 0 16781312 2 18446604451404639104 0 0 -1 6 0 0 0
[pid=4149/tid=4171] ppid=4147 vsize=738340 CPUtime=0 cores=0-7
/proc/4149/task/4171/stat : 4171 (python) S 4147 4149 3941 0 -1 4202560 100 936 0 0 0 0 0 0 15 0 35 0 668124648 756060160 19187 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562361265 0 0 16781312 2 18446604451404639104 0 0 -1 0 0 0 0
[pid=4149/tid=4174] ppid=4147 vsize=738340 CPUtime=0 cores=0-7
/proc/4149/task/4174/stat : 4174 (python) S 4147 4149 3941 0 -1 4202560 50 936 0 0 0 0 0 0 15 0 35 0 668124651 756060160 19187 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562361265 0 0 16781312 2 18446604451404639104 0 0 -1 0 0 0 0
[pid=4149/tid=4177] ppid=4147 vsize=738340 CPUtime=0 cores=0-7
/proc/4149/task/4177/stat : 4177 (python) S 4147 4149 3941 0 -1 4202560 63 936 0 0 0 0 0 0 15 0 35 0 668124654 756060160 19187 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562361265 0 0 16781312 2 18446604451404639104 0 0 -1 0 0 0 0
[pid=4149/tid=4180] ppid=4147 vsize=738340 CPUtime=0 cores=0-7
/proc/4149/task/4180/stat : 4180 (python) S 4147 4149 3941 0 -1 4202560 51 936 0 0 0 0 0 0 18 0 35 0 668124658 756060160 19187 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562361265 0 0 16781312 2 18446604451404639104 0 0 -1 7 0 0 0
[pid=4149/tid=4183] ppid=4147 vsize=738340 CPUtime=0 cores=0-7
/proc/4149/task/4183/stat : 4183 (python) S 4147 4149 3941 0 -1 4202560 52 936 0 0 0 0 0 0 18 0 35 0 668124661 756060160 19187 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562361265 0 0 16781312 2 18446604451404639104 0 0 -1 1 0 0 0
[pid=4149/tid=4186] ppid=4147 vsize=738340 CPUtime=0 cores=0-7
/proc/4149/task/4186/stat : 4186 (python) S 4147 4149 3941 0 -1 4202560 46 936 0 0 0 0 0 0 18 0 35 0 668124664 756060160 19187 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562361265 0 0 16781312 2 18446604451404639104 0 0 -1 1 0 0 0
[pid=4149/tid=4201] ppid=4147 vsize=738340 CPUtime=0 cores=0-7
/proc/4149/task/4201/stat : 4201 (python) S 4147 4149 3941 0 -1 4202560 47 936 0 0 0 0 0 0 18 0 35 0 668144213 756060160 19187 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562361265 0 0 16781312 2 18446604451404639104 0 0 -1 6 0 0 0
[pid=4149/tid=4212] ppid=4147 vsize=738340 CPUtime=0 cores=0-7
/proc/4149/task/4212/stat : 4212 (python) S 4147 4149 3941 0 -1 4202560 56 936 0 0 0 0 0 0 18 0 35 0 668154334 756060160 19187 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562361265 0 0 16781312 2 18446604451404639104 0 0 -1 6 0 0 0
[pid=4149/tid=4215] ppid=4147 vsize=738340 CPUtime=0 cores=0-7
/proc/4149/task/4215/stat : 4215 (python) S 4147 4149 3941 0 -1 4202560 34 936 0 0 0 0 0 0 22 0 35 0 668154338 756060160 19187 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562361265 0 0 16781312 2 18446604451404639104 0 0 -1 4 0 0 0
[pid=4149/tid=4221] ppid=4147 vsize=738340 CPUtime=0 cores=0-7
/proc/4149/task/4221/stat : 4221 (python) S 4147 4149 3941 0 -1 4202560 44 936 0 0 0 0 0 0 20 0 35 0 668163880 756060160 19187 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562361265 0 0 16781312 2 18446604451404639104 0 0 -1 1 0 0 0
[pid=4149/tid=4224] ppid=4147 vsize=738340 CPUtime=0 cores=0-7
/proc/4149/task/4224/stat : 4224 (python) S 4147 4149 3941 0 -1 4202560 39 936 0 0 0 0 0 0 19 0 35 0 668163982 756060160 19187 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562361265 0 0 16781312 2 18446604451404639104 0 0 -1 4 0 0 0
[pid=4149/tid=4228] ppid=4147 vsize=738340 CPUtime=0 cores=0-7
/proc/4149/task/4228/stat : 4228 (python) S 4147 4149 3941 0 -1 4202560 37 936 0 0 0 0 0 0 20 0 35 0 668164367 756060160 19187 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562361265 0 0 16781312 2 18446604451404639104 0 0 -1 4 0 0 0
[pid=4149/tid=4231] ppid=4147 vsize=738340 CPUtime=0 cores=0-7
/proc/4149/task/4231/stat : 4231 (python) S 4147 4149 3941 0 -1 4202560 35 936 0 0 0 0 0 0 20 0 35 0 668164681 756060160 19187 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562361265 0 0 16781312 2 18446604451404639104 0 0 -1 4 0 0 0
[pid=4149/tid=4237] ppid=4147 vsize=738340 CPUtime=0 cores=0-7
/proc/4149/task/4237/stat : 4237 (python) S 4147 4149 3941 0 -1 4202560 35 936 0 0 0 0 0 0 20 0 35 0 668173912 756060160 19187 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562361265 0 0 16781312 2 18446604451404639104 0 0 -1 1 0 0 0
[pid=4149/tid=4241] ppid=4147 vsize=738340 CPUtime=0 cores=0-7
/proc/4149/task/4241/stat : 4241 (python) S 4147 4149 3941 0 -1 4202560 34 936 0 0 0 0 0 0 20 0 35 0 668174008 756060160 19187 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562361265 0 0 16781312 2 18446604451404639104 0 0 -1 2 0 0 0
[pid=4149/tid=4246] ppid=4147 vsize=738340 CPUtime=0 cores=0-7
/proc/4149/task/4246/stat : 4246 (python) S 4147 4149 3941 0 -1 4202560 44 936 0 0 0 0 0 0 18 0 35 0 668174398 756060160 19187 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562361265 0 0 16781312 2 18446604451404639104 0 0 -1 4 0 0 0
[pid=4149/tid=4253] ppid=4147 vsize=738340 CPUtime=0 cores=0-7
/proc/4149/task/4253/stat : 4253 (python) S 4147 4149 3941 0 -1 4202560 34 936 0 0 0 0 0 0 20 0 35 0 668183030 756060160 19187 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562361265 0 0 16781312 2 18446604451404639104 0 0 -1 1 0 0 0
[pid=4149/tid=4256] ppid=4147 vsize=738340 CPUtime=0 cores=0-7
/proc/4149/task/4256/stat : 4256 (python) S 4147 4149 3941 0 -1 4202560 38 936 0 0 0 0 0 0 18 0 35 0 668183547 756060160 19187 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562361265 0 0 16781312 2 18446604451404639104 0 0 -1 4 0 0 0
[pid=4149/tid=4259] ppid=4147 vsize=738340 CPUtime=0 cores=0-7
/proc/4149/task/4259/stat : 4259 (python) S 4147 4149 3941 0 -1 4202560 35 936 0 0 0 0 0 0 19 0 35 0 668183734 756060160 19187 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562361265 0 0 16781312 2 18446604451404639104 0 0 -1 4 0 0 0
[pid=4149/tid=4263] ppid=4147 vsize=738340 CPUtime=0 cores=0-7
/proc/4149/task/4263/stat : 4263 (python) S 4147 4149 3941 0 -1 4202560 46 936 0 0 0 0 0 0 18 0 35 0 668183954 756060160 19187 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562361265 0 0 16781312 2 18446604451404639104 0 0 -1 4 0 0 0
[pid=4149/tid=4267] ppid=4147 vsize=738340 CPUtime=0 cores=0-7
/proc/4149/task/4267/stat : 4267 (python) S 4147 4149 3941 0 -1 4202560 47 936 0 0 0 0 0 0 18 0 35 0 668184040 756060160 19187 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562361265 0 0 16781312 2 18446604451404639104 0 0 -1 4 0 0 0
[pid=4149/tid=4270] ppid=4147 vsize=738340 CPUtime=0 cores=0-7
/proc/4149/task/4270/stat : 4270 (python) S 4147 4149 3941 0 -1 4202560 32 936 0 0 0 0 0 0 19 0 35 0 668184346 756060160 19187 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562361265 0 0 16781312 2 18446604451404639104 0 0 -1 4 0 0 0
[pid=4149/tid=4276] ppid=4147 vsize=738340 CPUtime=0 cores=0-7
/proc/4149/task/4276/stat : 4276 (python) S 4147 4149 3941 0 -1 4202560 33 936 0 0 0 0 0 0 19 0 35 0 668193270 756060160 19187 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562361265 0 0 16781312 2 18446604451404639104 0 0 -1 4 0 0 0
[pid=4149/tid=4281] ppid=4147 vsize=738340 CPUtime=0 cores=0-7
/proc/4149/task/4281/stat : 4281 (python) S 4147 4149 3941 0 -1 4202560 31 936 0 0 0 0 0 0 20 0 35 0 668193768 756060160 19187 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562361265 0 0 16781312 2 18446604451404639104 0 0 -1 4 0 0 0
[pid=4149/tid=4286] ppid=4147 vsize=738340 CPUtime=0 cores=0-7
/proc/4149/task/4286/stat : 4286 (python) S 4147 4149 3941 0 -1 4202560 43 936 0 0 0 0 0 0 18 0 35 0 668194380 756060160 19187 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562361265 0 0 16781312 2 18446604451404639104 0 0 -1 4 0 0 0
[pid=4149/tid=4292] ppid=4147 vsize=738340 CPUtime=0 cores=0-7
/proc/4149/task/4292/stat : 4292 (python) S 4147 4149 3941 0 -1 4202560 31 936 0 0 0 0 0 0 18 0 35 0 668203305 756060160 19187 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562361265 0 0 16781312 2 18446604451404639104 0 0 -1 1 0 0 0
[pid=4149/tid=4296] ppid=4147 vsize=738340 CPUtime=0 cores=0-7
/proc/4149/task/4296/stat : 4296 (python) S 4147 4149 3941 0 -1 4202560 36 936 0 0 0 0 0 0 19 0 35 0 668203608 756060160 19187 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562361265 0 0 16781312 2 18446604451404639104 0 0 -1 4 0 0 0
[pid=4149/tid=4304] ppid=4147 vsize=738340 CPUtime=0 cores=0-7
/proc/4149/task/4304/stat : 4304 (python) S 4147 4149 3941 0 -1 4202560 37 936 0 0 0 0 0 0 20 0 35 0 668213366 756060160 19187 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562361265 0 0 16781312 2 18446604451404639104 0 0 -1 4 0 0 0
[pid=4149/tid=4312] ppid=4147 vsize=738340 CPUtime=0 cores=0-7
/proc/4149/task/4312/stat : 4312 (python) S 4147 4149 3941 0 -1 4202560 42 936 0 0 0 0 0 0 18 0 35 0 668223292 756060160 19187 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562361265 0 0 16781312 2 18446604451404639104 0 0 -1 1 0 0 0
[pid=4149/tid=4316] ppid=4147 vsize=738340 CPUtime=0 cores=0-7
/proc/4149/task/4316/stat : 4316 (python) S 4147 4149 3941 0 -1 4202560 33 936 0 0 0 0 0 0 20 0 35 0 668224196 756060160 19187 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562361265 0 0 16781312 2 18446604451404639104 0 0 -1 4 0 0 0
[pid=4149/tid=4327] ppid=4147 vsize=738340 CPUtime=0 cores=0-7
/proc/4149/task/4327/stat : 4327 (python) S 4147 4149 3941 0 -1 4202560 36 936 0 0 0 0 0 0 18 0 35 0 668242975 756060160 19187 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562361265 0 0 16781312 2 18446604451404639104 0 0 -1 1 0 0 0
[pid=4149/tid=4336] ppid=4147 vsize=738340 CPUtime=0 cores=0-7
/proc/4149/task/4336/stat : 4336 (python) S 4147 4149 3941 0 -1 4202560 33 936 0 0 0 0 0 0 20 0 35 0 668261994 756060160 19187 33554432000 4194304 5683644 140733438452224 18446744073709551615 271562361265 0 0 16781312 2 18446604451404639104 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 10948.2
Current children cumulated vsize (KiB) 32367524

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 4149 and gives
#  childrusage.ru_utime.tv_sec=3027
#  childrusage.ru_utime.tv_usec=430760
#  childrusage.ru_stime.tv_sec=15
#  childrusage.ru_stime.tv_usec=852590
# CPU time returned by wait4() is 3043.28
# while last known CPU time is 10948.2
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1385.02
CPU time (s): 10948.2
CPU user time (s): 10932.3
CPU system time (s): 15.94
CPU usage (%): 790.473
Max. virtual memory (cumulated for all children) (KiB): 32367524

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 3027.43
system time used= 15.8526
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 173363
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 17876
involuntary context switches= 21226

runsolver used 14.4918 second user time and 36.2095 second system time

The end

Launcher Data 

Begin job on node133 at 2011-05-16 19:00:57
IDJOB=3385340
IDBENCH=82605
IDSOLVER=1870
FILE ID=node133/3385340-1305565256
RUNJOBID= node133-1305564962-3955
PBS_JOBID= 13350117
Free space on /tmp= 73420 MiB

SOLVER NAME= borg-sat 11.03.02-par-random (fixed)
BENCH NAME= SAT11/random/large/unif-k3-r4.2-v50000-c210000-S758437099-099-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3385340-1305565256/watcher-3385340-1305565256 -o /tmp/evaluation-result-3385340-1305565256/solver-3385340-1305565256 -C 40000 -W 5100 -M 31000  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3385340-1305565256.cnf -seed 6825901 -budget 40000 -cores 8

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 94d10ca78c6cee5b0badfb96b777ac08
RANDOM SEED=6825901

node133.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.834
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.834
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.834
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.834
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 6741.12
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.834
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.834
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.834
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.834
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      18702348 kB
Buffers:        319268 kB
Cached:        4517144 kB
SwapCached:       7676 kB
Active:       10634256 kB
Inactive:      3310640 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      18702348 kB
SwapTotal:    67111528 kB
SwapFree:     67056788 kB
Dirty:           35876 kB
Writeback:           0 kB
AnonPages:     9107480 kB
Mapped:          28464 kB
Slab:           200940 kB
PageTables:      35624 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 64708084 kB
VmallocTotal: 34359738367 kB
VmallocUsed:     33868 kB
VmallocChunk: 34359703231 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73388 MiB
End job on node133 at 2011-05-16 19:24:04