Trace number 3321832

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-random? (MO) 18619.2 2348.62

General information on the benchmark

Namerandom/large/
unif-k3-r4.2-v35000-c147000-S970100151-015-UNKNOWN.cnf
MD5SUM1ba3e47e885ea3862e5726ce64a96a9c
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark152.918
Satisfiable
(Un)Satisfiability was proved
Number of variables35000
Number of clauses147000
Sum of the clauses size441000
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 3147000
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.35/0.67	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.35/0.67	c INFO: solving HOME/instance-3321832-1304368028.cnf
0.35/0.67	c INFO: computing task features
0.84/1.18	c INFO: computed variable-clause graph statistics
0.92/1.20	c INFO: computed clause balance statistics
3.42/3.76	c INFO: computed variable balance statistics
5.52/5.85	c INFO: computed small-clause counts
5.52/5.86	c INFO: computed Horn-clause counts
7.12/7.42	c INFO: computed variable graph statistics
7.12/7.42	c INFO: collected features for HOME/instance-3321832-1304368028.cnf in 6.73 s
7.12/7.46	c INFO: running hybridGM3@0 for 97 with 39993 remaining (b = 0.36)
7.12/7.49	c INFO: running hybridGM3@0 for 97 with 39993 remaining (b = 0.22)
7.22/7.54	c INFO: running hybridGM3@0 for 97 with 39993 remaining (b = 0.14)
7.22/7.58	c INFO: running hybridGM3@0 for 194 with 39993 remaining (b = 0.18)
7.56/7.62	c INFO: running hybridGM3@0 for 194 with 39993 remaining (b = 0.14)
7.56/7.66	c INFO: running hybridGM3@0 for 194 with 39993 remaining (b = 0.11)
7.56/7.70	c INFO: running TNM@0 for 97 with 39992 remaining (b = 0.04)
8.00/7.74	c INFO: running hybridGM3@0 for 194 with 39992 remaining (b = 0.07)
803.79/107.88	c INFO: running hybridGM3@0 for 97 with 39892 remaining (b = 0.03)
805.31/108.02	c INFO: running hybridGM3@0 for 194 with 39892 remaining (b = 0.06)
805.31/108.07	c INFO: running hybridGM3@0 for 97 with 39892 remaining (b = 0.03)
838.10/112.15	c INFO: running hybridGM3@0 for 97 with 39888 remaining (b = 0.03)
1556.25/202.12	c INFO: running hybridGM3@97 for 97 with 39798 remaining (b = 0.03)
1565.03/203.29	c INFO: running hybridGM3@102 for 97 with 39797 remaining (b = 0.03)
1571.42/204.07	c INFO: running hybridGM3@0 for 97 with 39796 remaining (b = 0.03)
1572.21/204.19	c INFO: running hybridGM3@0 for 97 with 39796 remaining (b = 0.03)
1575.41/204.57	c INFO: running hybridGM3@100 for 97 with 39796 remaining (b = 0.03)
1597.67/207.37	c INFO: running hybridGM3@93 for 97 with 39793 remaining (b = 0.03)
1620.08/210.18	c INFO: running hybridGM3@0 for 97 with 39790 remaining (b = 0.03)
2331.78/299.30	c INFO: running hybridGM3@0 for 194 with 39701 remaining (b = 0.04)
2332.50/299.43	c INFO: running hybridGM3@0 for 194 with 39701 remaining (b = 0.03)
2334.93/299.80	c INFO: running hybridGM3@0 for 194 with 39700 remaining (b = 0.03)
2355.70/302.38	c INFO: running TNM@0 for 97 with 39698 remaining (b = 0.01)
2373.17/304.54	c INFO: running hybridGM3@192 for 97 with 39696 remaining (b = 0.01)
2373.17/304.59	c INFO: running hybridGM3@99 for 97 with 39696 remaining (b = 0.01)
2380.35/305.43	c INFO: running hybridGM3@95 for 97 with 39695 remaining (b = 0.01)
2405.89/308.62	c INFO: running hybridGM3@298 for 97 with 39692 remaining (b = 0.01)
2876.64/367.63	c INFO: running hybridGM3@292 for 97 with 39633 remaining (b = 0.01)
3146.98/402.69	c INFO: running TNM@0 for 1940 with 39598 remaining (b = 0.21)
3147.68/402.80	c INFO: running TNM@0 for 97 with 39597 remaining (b = 0.01)
3165.96/405.02	c INFO: running hybridGM3@356 for 97 with 39595 remaining (b = 0.01)
3165.96/405.07	c INFO: running hybridGM3@388 for 97 with 39595 remaining (b = 0.01)
3643.16/464.89	c INFO: running TNM@0 for 97 with 39535 remaining (b = 0.01)
3892.09/496.07	c INFO: running hybridGM3@94 for 97 with 39504 remaining (b = 0.01)
3895.22/496.40	c INFO: running hybridGM3@99 for 97 with 39504 remaining (b = 0.01)
3925.54/500.27	c INFO: running hybridGM3@392 for 97 with 39500 remaining (b = 0.01)
3939.09/501.94	c INFO: running hybridGM3@95 for 97 with 39498 remaining (b = 0.01)
3947.86/503.07	c INFO: running hybridGM3@452 for 97 with 39497 remaining (b = 0.01)
3950.23/503.35	c INFO: running hybridGM3@485 for 97 with 39497 remaining (b = 0.01)
4443.22/565.16	c INFO: running hybridGM3@0 for 194 with 39435 remaining (b = 0.02)
4443.93/565.25	c INFO: running adaptg2wsat2009++@0 for 194 with 39435 remaining (b = 0.02)
4689.06/597.19	c INFO: running hybridGM3@193 for 97 with 39403 remaining (b = 0.01)
4692.20/597.54	c INFO: running TNM@0 for 97 with 39403 remaining (b = 0.01)
4692.93/597.67	c INFO: running TNM@0 for 97 with 39403 remaining (b = 0.01)
4730.35/602.35	c INFO: running hybridGM3@0 for 194 with 39398 remaining (b = 0.02)
4748.68/604.63	c INFO: running TNM@0 for 97 with 39396 remaining (b = 0.01)
5448.38/692.46	c INFO: running TNM@0 for 97 with 39308 remaining (b = 0.01)
5483.43/696.85	c INFO: running TNM@0 for 97 with 39303 remaining (b = 0.01)
5492.16/697.99	c INFO: running TNM@195 for 1843 with 39302 remaining (b = 0.12)
5540.03/703.96	c INFO: running hybridGM3@0 for 194 with 39296 remaining (b = 0.01)
6001.34/761.88	c INFO: running hybridGM3@64 for 97 with 39238 remaining (b = 0.01)
6088.97/772.81	c INFO: running adaptg2wsat2009++@94 for 97 with 39227 remaining (b = 0.01)
6247.49/792.78	c INFO: running adaptg2wsat2009++@0 for 194 with 39207 remaining (b = 0.01)
6282.46/797.17	c INFO: running hybridGM3@0 for 194 with 39203 remaining (b = 0.01)
6297.55/799.02	c INFO: running hybridGM3@98 for 97 with 39201 remaining (b = 0.01)
6807.78/863.20	c INFO: running adaptg2wsat2009++@0 for 194 with 39137 remaining (b = 0.01)
6847.59/868.12	c INFO: running hybridGM3@196 for 291 with 39132 remaining (b = 0.02)
7103.88/900.34	c INFO: running FH@0 for 97 with 39100 remaining (b = 0.01)
7146.03/905.69	c INFO: running hybridGM3@196 for 291 with 39095 remaining (b = 0.02)
7848.07/993.87	c INFO: running hybridGM3@195 for 291 with 39006 remaining (b = 0.01)
7899.81/1000.38	c INFO: running adaptg2wsat2009++@100 for 97 with 39000 remaining (b = 0.01)
7902.21/1000.68	c INFO: running adaptg2wsat2009++@0 for 194 with 39000 remaining (b = 0.01)
8372.57/1059.74	c INFO: running hybridGM3@194 for 291 with 38941 remaining (b = 0.01)
8658.34/1095.69	c INFO: running TNM@0 for 97 with 38905 remaining (b = 0.00)
9201.12/1163.87	c INFO: running hybridGM3@296 for 194 with 38836 remaining (b = 0.01)
9449.34/1195.06	c INFO: running hybridGM3@195 for 291 with 38805 remaining (b = 0.01)
9466.84/1197.23	c INFO: running TNM@295 for 1746 with 38803 remaining (b = 0.09)
9500.28/1201.44	c INFO: running hybridGM3@392 for 97 with 38799 remaining (b = 0.01)
10234.82/1293.76	c INFO: running hybridGM3@394 for 97 with 38707 remaining (b = 0.01)
10258.71/1296.76	c INFO: running hybridGM3@290 for 291 with 38704 remaining (b = 0.01)
10725.06/1355.39	c INFO: running hybridGM3@395 for 97 with 38645 remaining (b = 0.01)
10822.11/1367.52	c INFO: running hybridGM3@0 for 194 with 38633 remaining (b = 0.01)
10993.15/1389.09	c INFO: running adaptg2wsat2009++@94 for 97 with 38611 remaining (b = 0.00)
11483.94/1450.71	c INFO: running adaptg2wsat2009++@0 for 194 with 38550 remaining (b = 0.01)
11755.00/1484.84	c INFO: running hybridGM3@391 for 97 with 38515 remaining (b = 0.01)
11767.72/1486.42	c INFO: running adaptg2wsat2009++@190 for 97 with 38514 remaining (b = 0.00)
12386.27/1564.25	c INFO: running hybridGM3@95 for 97 with 38436 remaining (b = 0.00)
12560.32/1586.19	c INFO: running hybridGM3@263 for 291 with 38414 remaining (b = 0.01)
12572.98/1587.75	c INFO: running adaptg2wsat2009++@385 for 291 with 38413 remaining (b = 0.01)
12611.14/1592.53	c INFO: running hybridGM3@481 for 97 with 38408 remaining (b = 0.01)
13046.79/1647.32	c INFO: running NCVWr@0 for 485 with 38353 remaining (b = 0.02)
13192.25/1665.61	c INFO: running FH@0 for 97 with 38335 remaining (b = 0.00)
13368.69/1687.87	c INFO: running hybridGM3@198 for 291 with 38312 remaining (b = 0.01)
13989.57/1765.97	c INFO: running hybridGM3@195 for 291 with 38234 remaining (b = 0.01)
14860.97/1875.53	c INFO: running adaptg2wsat2009++@681 for 97 with 38125 remaining (b = 0.00)
14864.07/1875.99	c INFO: running hybridGM3@364 for 194 with 38124 remaining (b = 0.01)
15666.21/1976.89	c INFO: running adaptg2wsat2009++@969 for 97 with 38023 remaining (b = 0.01)
15672.57/1977.68	c INFO: running hybridGM3@298 for 194 with 38023 remaining (b = 0.01)
16293.36/2055.77	c INFO: running hybridGM3@482 for 97 with 37944 remaining (b = 0.01)
16424.51/2072.24	c INFO: running hybridGM3@196 for 291 with 37928 remaining (b = 0.01)
16434.90/2073.57	c INFO: running hybridGM3@465 for 97 with 37927 remaining (b = 0.01)
16944.39/2137.61	c INFO: running NCVWr@101 for 388 with 37863 remaining (b = 0.02)
17051.68/2151.11	c INFO: running TNM@100 for 1455 with 37849 remaining (b = 0.06)
17193.12/2168.91	c INFO: running hybridGM3@0 for 194 with 37831 remaining (b = 0.01)
17243.20/2175.27	c INFO: running FH@199 for 194 with 37825 remaining (b = 0.01)
18618.29/2348.39	c INFO: running NCVWr@0 for 485 with 37652 remaining (b = 0.02)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3321832-1304368028/watcher-3321832-1304368028 -o /tmp/evaluation-result-3321832-1304368028/solver-3321832-1304368028 -C 40000 -W 5100 -M 31000 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3321832-1304368028.cnf -seed 1174201737 -budget 40000 -cores 8 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.25 8.48 6.84 1/269 2904
/proc/meminfo: memFree=27633668/32951132 swapFree=65145212/67111528
[pid=2904] ppid=2902 vsize=15184 CPUtime=0 cores=0-7
/proc/2904/stat : 2904 (python) S 2902 2904 2328 0 -1 4202496 712 0 0 0 0 0 0 0 18 0 1 0 548402522 15548416 603 33554432000 4194304 5683644 140733781901008 18446744073709551615 210647144448 0 2147483391 16781312 2 18446744071699109054 0 0 17 2 0 0 0
/proc/2904/statm: 3796 603 319 364 0 283 0

[startup+0.0311309 s]
/proc/loadavg: 8.25 8.48 6.84 1/269 2904
/proc/meminfo: memFree=27633668/32951132 swapFree=65145212/67111528
[pid=2904] ppid=2902 vsize=16392 CPUtime=0.01 cores=0-7
/proc/2904/stat : 2904 (python) S 2902 2904 2328 0 -1 4202496 1165 0 0 0 1 0 0 0 17 0 1 0 548402522 16785408 1029 33554432000 4194304 5683644 140733781901008 18446744073709551615 210647144448 0 2147483391 16781312 2 18446744071699109054 0 0 17 2 0 0 0
/proc/2904/statm: 4098 1029 414 364 0 585 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 16392

[startup+0.101124 s]
/proc/loadavg: 8.25 8.48 6.84 1/269 2904
/proc/meminfo: memFree=27633668/32951132 swapFree=65145212/67111528
[pid=2904] ppid=2902 vsize=89764 CPUtime=0.06 cores=0-7
/proc/2904/stat : 2904 (python) S 2902 2904 2328 0 -1 4202496 2672 0 0 0 4 2 0 0 17 0 1 0 548402522 91918336 2387 33554432000 4194304 5683644 140733781901008 18446744073709551615 210647144448 0 2147483391 16781312 2 18446744071699109054 0 0 17 2 0 0 0
/proc/2904/statm: 22441 2387 750 364 0 1621 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 89764

[startup+0.301088 s]
/proc/loadavg: 8.25 8.48 6.84 1/269 2904
/proc/meminfo: memFree=27633668/32951132 swapFree=65145212/67111528
[pid=2904] ppid=2902 vsize=135748 CPUtime=0.18 cores=0-7
/proc/2904/stat : 2904 (python) S 2902 2904 2328 0 -1 4202496 5968 933 1 0 13 5 0 0 18 0 1 0 548402522 139005952 4151 33554432000 4194304 5683644 140733781901008 18446744073709551615 210647142405 0 2147483391 16781312 2 18446744071699109054 0 0 17 2 0 0 0
/proc/2904/statm: 33937 4151 1032 364 0 3123 0
Current children cumulated CPU time (s) 0.18
Current children cumulated vsize (KiB) 135748

[startup+0.701011 s]
/proc/loadavg: 8.25 8.48 6.84 1/269 2904
/proc/meminfo: memFree=27633668/32951132 swapFree=65145212/67111528
[pid=2904] ppid=2902 vsize=283080 CPUtime=0.42 cores=0-7
/proc/2904/stat : 2904 (python) R 2902 2904 2328 0 -1 4202496 10415 933 1 0 31 11 0 0 17 0 1 0 548402522 289873920 8093 33554432000 4194304 5683644 140733781901008 18446744073709551615 4952096 0 0 16781312 2 0 0 0 17 2 0 0 0
/proc/2904/statm: 70770 8093 1761 364 0 6694 0
Current children cumulated CPU time (s) 0.42
Current children cumulated vsize (KiB) 283080

[startup+1.50086 s]
/proc/loadavg: 8.25 8.48 6.84 2/270 2910
/proc/meminfo: memFree=27589268/32951132 swapFree=65145212/67111528
[pid=2904] ppid=2902 vsize=312968 CPUtime=1.22 cores=0-7
/proc/2904/stat : 2904 (python) R 2902 2904 2328 0 -1 4202496 21970 933 1 0 109 13 0 0 18 0 1 0 548402522 320479232 15627 33554432000 4194304 5683644 140733781901008 18446744073709551615 4623696 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/2904/statm: 78242 15627 1826 364 0 14166 0
Current children cumulated CPU time (s) 1.22
Current children cumulated vsize (KiB) 312968

[startup+3.10059 s]
/proc/loadavg: 8.25 8.48 6.84 3/270 2910
/proc/meminfo: memFree=27577172/32951132 swapFree=65145212/67111528
[pid=2904] ppid=2902 vsize=312968 CPUtime=2.82 cores=0-7
/proc/2904/stat : 2904 (python) R 2902 2904 2328 0 -1 4202496 21970 933 1 0 269 13 0 0 21 0 1 0 548402522 320479232 15627 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970452 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/2904/statm: 78242 15627 1826 364 0 14166 0
Current children cumulated CPU time (s) 2.82
Current children cumulated vsize (KiB) 312968

[startup+6.30105 s]
/proc/loadavg: 7.83 8.39 6.81 2/270 2910
/proc/meminfo: memFree=27578104/32951132 swapFree=65145212/67111528
[pid=2904] ppid=2902 vsize=312968 CPUtime=6.02 cores=0-7
/proc/2904/stat : 2904 (python) R 2902 2904 2328 0 -1 4202496 21977 933 1 0 589 13 0 0 25 0 1 0 548402522 320479232 15634 33554432000 4194304 5683644 140733781901008 18446744073709551615 47008278202508 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/2904/statm: 78242 15634 1833 364 0 14166 0
Current children cumulated CPU time (s) 6.02
Current children cumulated vsize (KiB) 312968

[startup+12.7019 s]
/proc/loadavg: 8.08 8.44 6.84 9/294 2936
/proc/meminfo: memFree=27305864/32951132 swapFree=65145212/67111528
[pid=2904] ppid=2902 vsize=457832 CPUtime=7.47 cores=0-7
/proc/2904/stat : 2904 (python) S 2902 2904 2328 0 -1 4202496 28337 933 1 0 731 16 0 0 23 0 9 0 548402522 468819968 15185 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658973131 0 0 16781312 2 18446744073709551615 0 0 17 0 0 0 0
/proc/2904/statm: 114458 15185 1889 364 0 50355 0
[pid=2911] ppid=2904 vsize=310260 CPUtime=0 cores=0-7
/proc/2911/stat : 2911 (python) S 2904 2904 2328 0 -1 4202560 901 0 0 0 0 0 0 0 15 0 1 0 548403268 317706240 13494 33554432000 4194304 5683644 140733781901008 18446744073709551615 210647166479 0 0 16781312 514 16652807019526324484 0 0 17 7 0 0 0
/proc/2911/statm: 77565 13494 376 364 0 13483 0
[pid=2913] ppid=2911 vsize=268420 CPUtime=4.27 cores=0-7
/proc/2913/stat : 2913 (hybridGM3) R 2911 2913 2913 0 -1 4202496 8084 0 0 0 426 1 0 0 25 0 1 0 548403268 274862080 7822 33554432000 134512640 135204385 4287466848 18446744073709551615 134537510 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/2913/statm: 67105 7822 64 169 0 66935 0
[pid=2914] ppid=2904 vsize=320516 CPUtime=0 cores=0-7
/proc/2914/stat : 2914 (python) S 2904 2904 2328 0 -1 4202560 918 0 0 0 0 0 0 0 15 0 1 0 548403272 328208384 13492 33554432000 4194304 5683644 140733781901008 18446744073709551615 210647166534 0 0 16781312 514 16652810867817021699 0 0 17 2 0 0 0
/proc/2914/statm: 80129 13492 370 364 0 16044 0
[pid=2916] ppid=2914 vsize=268424 CPUtime=4.92 cores=0-7
/proc/2916/stat : 2916 (hybridGM3) R 2914 2916 2916 0 -1 4202496 8082 0 0 0 481 11 0 0 25 0 1 0 548403272 274866176 7821 33554432000 134512640 135204385 4294762224 18446744073709551615 134546082 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/2916/statm: 67106 7821 64 169 0 66936 0
[pid=2917] ppid=2904 vsize=330772 CPUtime=0 cores=0-7
/proc/2917/stat : 2917 (python) S 2904 2904 2328 0 -1 4202560 934 0 0 0 0 0 0 0 15 0 1 0 548403276 338710528 13496 33554432000 4194304 5683644 140733781901008 18446744073709551615 210647166534 0 0 16781312 514 1999389056760250627 0 0 17 7 0 0 0
/proc/2917/statm: 82693 13496 370 364 0 18605 0
[pid=2919] ppid=2917 vsize=268420 CPUtime=5.15 cores=0-7
/proc/2919/stat : 2919 (hybridGM3) R 2917 2919 2919 0 -1 4202496 8081 0 0 0 514 1 0 0 25 0 1 0 548403276 274862080 7819 33554432000 134512640 135204385 4290199296 18446744073709551615 134537383 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/2919/statm: 67105 7819 64 169 0 66935 0
[pid=2920] ppid=2904 vsize=341028 CPUtime=0 cores=0-7
/proc/2920/stat : 2920 (python) S 2904 2904 2328 0 -1 4202560 946 0 0 0 0 0 0 0 15 0 1 0 548403280 349212672 13499 33554432000 4194304 5683644 140733781901008 18446744073709551615 210647166534 0 0 16781312 514 1999396753341645059 0 0 17 2 0 0 0
/proc/2920/statm: 85257 13499 369 364 0 21166 0
[pid=2922] ppid=2920 vsize=268424 CPUtime=4.71 cores=0-7
/proc/2922/stat : 2922 (hybridGM3) R 2920 2922 2922 0 -1 4202496 8088 0 0 0 470 1 0 0 25 0 1 0 548403280 274866176 7827 33554432000 134512640 135204385 4286666112 18446744073709551615 134537290 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/2922/statm: 67106 7827 64 169 0 66936 0
[pid=2923] ppid=2904 vsize=351284 CPUtime=0 cores=0-7
/proc/2923/stat : 2923 (python) S 2904 2904 2328 0 -1 4202560 949 0 0 0 0 0 0 0 15 0 1 0 548403284 359714816 13504 33554432000 4194304 5683644 140733781901008 18446744073709551615 210647166534 0 0 16781312 514 4346021748341178625 0 0 17 1 0 0 0
/proc/2923/statm: 87821 13504 370 364 0 23727 0
[pid=2925] ppid=2923 vsize=268420 CPUtime=4.99 cores=0-7
/proc/2925/stat : 2925 (hybridGM3) R 2923 2925 2925 0 -1 4202496 8085 0 0 0 498 1 0 0 25 0 1 0 548403285 274862080 7821 33554432000 134512640 135204385 4288472160 18446744073709551615 134537154 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/2925/statm: 67105 7821 64 169 0 66935 0
[pid=2926] ppid=2904 vsize=427076 CPUtime=0 cores=0-7
/proc/2926/stat : 2926 (python) S 2904 2904 2328 0 -1 4202560 804 0 0 0 0 0 0 0 15 0 1 0 548403288 437325824 13682 33554432000 4194304 5683644 140733781901008 18446744073709551615 210647166534 0 0 16781312 514 4346017900050481411 0 0 17 1 0 0 0
/proc/2926/statm: 106769 13682 371 364 0 42672 0
[pid=2928] ppid=2926 vsize=268424 CPUtime=4.98 cores=0-7
/proc/2928/stat : 2928 (hybridGM3) R 2926 2928 2928 0 -1 4202496 8076 0 0 0 497 1 0 0 25 0 1 0 548403289 274866176 7822 33554432000 134512640 135204385 4286652992 18446744073709551615 134537272 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/2928/statm: 67106 7822 64 169 0 66936 0
[pid=2931] ppid=2904 vsize=437332 CPUtime=0 cores=0-7
/proc/2931/stat : 2931 (python) S 2904 2904 2328 0 -1 4202560 819 0 0 0 0 0 0 0 18 0 1 0 548403292 447827968 13658 33554432000 4194304 5683644 140733781901008 18446744073709551615 210647166534 0 0 16781312 514 4346014051759784196 0 0 17 1 0 0 0
/proc/2931/statm: 109333 13658 371 364 0 45233 0
[pid=2936] ppid=2931 vsize=465044 CPUtime=2.6 cores=0-7
/proc/2936/stat : 2936 (TNM) R 2931 2936 2936 0 -1 4202496 6006 0 0 0 245 15 0 0 25 0 1 0 548403301 476205056 5749 33554432000 134512640 135008748 4292565472 18446744073709551615 134527542 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/2936/statm: 116261 5749 46 122 0 115161 0
[pid=2933] ppid=2904 vsize=447588 CPUtime=0.32 cores=0-7
/proc/2933/stat : 2933 (python) S 2904 2904 2328 0 -1 4202560 858 0 0 0 13 19 0 0 18 0 1 0 548403297 458330112 13669 33554432000 4194304 5683644 140733781901008 18446744073709551615 210647166534 0 0 16781312 514 4346010203469086979 0 0 17 0 0 0 0
/proc/2933/statm: 111897 13669 377 364 0 47794 0
[pid=2935] ppid=2933 vsize=268420 CPUtime=4.78 cores=0-7
/proc/2935/stat : 2935 (hybridGM3) R 2933 2935 2935 0 -1 4202496 8075 0 0 0 478 0 0 0 25 0 1 0 548403297 274862080 7818 33554432000 134512640 135204385 4289469216 18446744073709551615 134547418 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/2935/statm: 67105 7818 64 169 0 66935 0
[pid=2904/tid=2912] ppid=2902 vsize=457832 CPUtime=0 cores=0-7
/proc/2904/task/2912/stat : 2912 (python) S 2902 2904 2328 0 -1 4202560 24 933 0 0 0 0 0 0 25 0 9 0 548403268 468819968 15185 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970033 0 0 16781312 2 18446604442507624512 0 0 -1 6 0 0 0
[pid=2904/tid=2915] ppid=2902 vsize=457832 CPUtime=0 cores=0-7
/proc/2904/task/2915/stat : 2915 (python) S 2902 2904 2328 0 -1 4202560 22 933 0 0 0 0 0 0 25 0 9 0 548403272 468819968 15185 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970033 0 0 16781312 2 18446604442507624512 0 0 -1 2 0 0 0
[pid=2904/tid=2918] ppid=2902 vsize=457832 CPUtime=0 cores=0-7
/proc/2904/task/2918/stat : 2918 (python) S 2902 2904 2328 0 -1 4202560 22 933 0 0 0 0 0 0 25 0 9 0 548403276 468819968 15185 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970033 0 0 16781312 2 18446604442507624512 0 0 -1 6 0 0 0
[pid=2904/tid=2921] ppid=2902 vsize=457832 CPUtime=0 cores=0-7
/proc/2904/task/2921/stat : 2921 (python) S 2902 2904 2328 0 -1 4202560 23 933 0 0 0 0 0 0 20 0 9 0 548403280 468819968 15185 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970033 0 0 16781312 2 18446604442507624512 0 0 -1 6 0 0 0
[pid=2904/tid=2924] ppid=2902 vsize=457832 CPUtime=0 cores=0-7
/proc/2904/task/2924/stat : 2924 (python) S 2902 2904 2328 0 -1 4202560 27 933 0 0 0 0 0 0 18 0 9 0 548403284 468819968 15185 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970033 0 0 16781312 2 18446604442507624512 0 0 -1 7 0 0 0
[pid=2904/tid=2927] ppid=2902 vsize=457832 CPUtime=0 cores=0-7
/proc/2904/task/2927/stat : 2927 (python) S 2902 2904 2328 0 -1 4202560 99 933 0 0 0 0 0 0 18 0 9 0 548403289 468819968 15185 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970033 0 0 16781312 2 18446604442507624512 0 0 -1 1 0 0 0
[pid=2904/tid=2932] ppid=2902 vsize=457832 CPUtime=0 cores=0-7
/proc/2904/task/2932/stat : 2932 (python) S 2902 2904 2328 0 -1 4202560 35 933 0 0 0 0 0 0 21 0 9 0 548403293 468819968 15185 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970033 0 0 16781312 2 18446604442507624512 0 0 -1 3 0 0 0
[pid=2904/tid=2934] ppid=2902 vsize=457832 CPUtime=0 cores=0-7
/proc/2904/task/2934/stat : 2934 (python) S 2902 2904 2328 0 -1 4202560 79 933 0 0 0 0 0 0 23 0 9 0 548403297 468819968 15185 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970033 0 0 16781312 2 18446604442507624512 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 44.19
Current children cumulated vsize (KiB) 5767684

[startup+25.5018 s]
/proc/loadavg: 8.06 8.41 6.85 9/294 2937
/proc/meminfo: memFree=27308352/32951132 swapFree=65145212/67111528
[pid=2904] ppid=2902 vsize=457832 CPUtime=7.47 cores=0-7
/proc/2904/stat : 2904 (python) S 2902 2904 2328 0 -1 4202496 28337 933 1 0 731 16 0 0 23 0 9 0 548402522 468819968 15185 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658973131 0 0 16781312 2 18446744073709551615 0 0 17 0 0 0 0
/proc/2904/statm: 114458 15185 1889 364 0 50355 0
[pid=2911] ppid=2904 vsize=310260 CPUtime=0.03 cores=0-7
/proc/2911/stat : 2911 (python) S 2904 2904 2328 0 -1 4202560 907 0 0 0 1 2 0 0 15 0 1 0 548403268 317706240 13494 33554432000 4194304 5683644 140733781901008 18446744073709551615 210647166479 0 0 16781312 514 17631539764597326084 0 0 17 7 0 0 0
/proc/2911/statm: 77565 13494 376 364 0 13483 0
[pid=2913] ppid=2911 vsize=268420 CPUtime=16.97 cores=0-7
/proc/2913/stat : 2913 (hybridGM3) R 2911 2913 2913 0 -1 4202496 8090 0 0 0 1694 3 0 0 25 0 1 0 548403268 274862080 7828 33554432000 134512640 135204385 4287466848 18446744073709551615 134547471 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/2913/statm: 67105 7828 64 169 0 66935 0
[pid=2914] ppid=2904 vsize=320516 CPUtime=0.02 cores=0-7
/proc/2914/stat : 2914 (python) S 2904 2904 2328 0 -1 4202560 923 0 0 0 1 1 0 0 15 0 1 0 548403272 328208384 13492 33554432000 4194304 5683644 140733781901008 18446744073709551615 210647166534 0 0 16781312 514 17631539764597326083 0 0 17 2 0 0 0
/proc/2914/statm: 80129 13492 370 364 0 16044 0
[pid=2916] ppid=2914 vsize=268424 CPUtime=17.67 cores=0-7
/proc/2916/stat : 2916 (hybridGM3) R 2914 2916 2916 0 -1 4202496 8083 0 0 0 1756 11 0 0 25 0 1 0 548403272 274866176 7822 33554432000 134512640 135204385 4294762224 18446744073709551615 134536744 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/2916/statm: 67106 7822 64 169 0 66936 0
[pid=2917] ppid=2904 vsize=330772 CPUtime=0.03 cores=0-7
/proc/2917/stat : 2917 (python) S 2904 2904 2328 0 -1 4202560 937 0 0 0 2 1 0 0 15 0 1 0 548403276 338710528 13496 33554432000 4194304 5683644 140733781901008 18446744073709551615 210647166534 0 0 16781312 514 17631539764597326083 0 0 17 7 0 0 0
/proc/2917/statm: 82693 13496 370 364 0 18605 0
[pid=2919] ppid=2917 vsize=268420 CPUtime=17.95 cores=0-7
/proc/2919/stat : 2919 (hybridGM3) R 2917 2919 2919 0 -1 4202496 8083 0 0 0 1794 1 0 0 25 0 1 0 548403276 274862080 7821 33554432000 134512640 135204385 4290199296 18446744073709551615 134547403 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/2919/statm: 67105 7821 64 169 0 66935 0
[pid=2920] ppid=2904 vsize=341028 CPUtime=0.03 cores=0-7
/proc/2920/stat : 2920 (python) S 2904 2904 2328 0 -1 4202560 953 0 0 0 2 1 0 0 15 0 1 0 548403280 349212672 13500 33554432000 4194304 5683644 140733781901008 18446744073709551615 210647166534 0 0 16781312 514 17631539764597326083 0 0 17 2 0 0 0
/proc/2920/statm: 85257 13500 370 364 0 21166 0
[pid=2922] ppid=2920 vsize=268424 CPUtime=17.51 cores=0-7
/proc/2922/stat : 2922 (hybridGM3) R 2920 2922 2922 0 -1 4202496 8088 0 0 0 1750 1 0 0 25 0 1 0 548403280 274866176 7827 33554432000 134512640 135204385 4286666112 18446744073709551615 134537455 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/2922/statm: 67106 7827 64 169 0 66936 0
[pid=2923] ppid=2904 vsize=351284 CPUtime=0.03 cores=0-7
/proc/2923/stat : 2923 (python) S 2904 2904 2328 0 -1 4202560 953 0 0 0 2 1 0 0 15 0 1 0 548403284 359714816 13504 33554432000 4194304 5683644 140733781901008 18446744073709551615 210647166534 0 0 16781312 514 17631539764597326081 0 0 17 1 0 0 0
/proc/2923/statm: 87821 13504 370 364 0 23727 0
[pid=2925] ppid=2923 vsize=268420 CPUtime=17.6 cores=0-7
/proc/2925/stat : 2925 (hybridGM3) R 2923 2925 2925 0 -1 4202496 8089 0 0 0 1759 1 0 0 25 0 1 0 548403285 274862080 7825 33554432000 134512640 135204385 4288472160 18446744073709551615 134536676 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/2925/statm: 67105 7825 64 169 0 66935 0

################
# More data... #
################

/proc/3035/task/3051/stat : 3051 (python) S 2904 2904 2328 0 -1 4202560 87 0 0 0 0 0 0 0 16 0 2 0 548501899 647368704 14005 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970033 0 0 16781312 514 18446604453823381824 0 0 -1 6 0 0 0
[pid=3041] ppid=2904 vsize=642452 CPUtime=2.14 cores=0-7
/proc/3041/stat : 3041 (python) S 2904 2904 2328 0 -1 4202560 1205 0 0 0 110 104 0 0 18 0 2 0 548488842 657870848 14005 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658973131 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/3041/statm: 160613 14005 401 364 0 96453 0
[pid=3043] ppid=3041 vsize=431704 CPUtime=875.9 cores=0-7
/proc/3043/stat : 3043 (adaptg2wsat2009) T 3041 3043 3043 0 -1 4202496 6266 0 0 0 87587 3 0 0 25 0 1 0 548488843 442064896 5983 33554432000 134512640 135107411 4289993072 18446744073709551615 134531942 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/3043/statm: 107926 5983 59 146 0 106802 0
[pid=3041/tid=3060] ppid=2904 vsize=642452 CPUtime=0 cores=0-7
/proc/3041/task/3060/stat : 3060 (python) S 2904 2904 2328 0 -1 4202560 94 0 0 0 0 0 0 0 18 0 2 0 548508486 657870848 14005 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970033 0 0 16781312 514 18446604453823384512 0 0 -1 1 0 0 0
[pid=3045] ppid=2904 vsize=652708 CPUtime=0.67 cores=0-7
/proc/3045/stat : 3045 (python) S 2904 2904 2328 0 -1 4202560 1207 0 0 0 32 35 0 0 18 0 2 0 548492556 668372992 14011 33554432000 4194304 5683644 140733781901008 18446744073709551615 210647166534 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/3045/statm: 163177 14011 401 364 0 99014 0
[pid=3047] ppid=3045 vsize=260404 CPUtime=272.84 cores=0-7
/proc/3047/stat : 3047 (FH) R 3045 3047 3047 0 -1 4202496 6185 0 0 0 27283 1 0 0 25 0 1 0 548492557 266653696 5899 33554432000 134512640 134936988 4286985744 18446744073709551615 134547124 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/3047/statm: 65101 5899 41 104 0 64605 0
[pid=3045/tid=3054] ppid=2904 vsize=652708 CPUtime=0 cores=0-7
/proc/3045/task/3054/stat : 3054 (python) S 2904 2904 2328 0 -1 4202560 91 0 0 0 0 0 0 0 19 0 2 0 548502580 668372992 14011 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970033 0 0 16781312 514 18446604453857707200 0 0 -1 1 0 0 0
[pid=3055] ppid=2904 vsize=662964 CPUtime=0.46 cores=0-7
/proc/3055/stat : 3055 (python) S 2904 2904 2328 0 -1 4202560 1250 0 0 0 23 23 0 0 19 0 2 0 548502590 678875136 14018 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658973131 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/3055/statm: 165741 14018 407 364 0 101575 0
[pid=3057] ppid=3055 vsize=431704 CPUtime=196.15 cores=0-7
/proc/3057/stat : 3057 (adaptg2wsat2009) T 3055 3057 3057 0 -1 4202496 6277 0 0 0 19613 2 0 0 25 0 1 0 548502591 442064896 5985 33554432000 134512640 135107411 4287909824 18446744073709551615 134531856 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/3057/statm: 107926 5985 59 146 0 106802 0
[pid=3055/tid=3069] ppid=2904 vsize=662964 CPUtime=0 cores=0-7
/proc/3055/task/3069/stat : 3069 (python) S 2904 2904 2328 0 -1 4202560 97 0 0 0 0 0 0 0 19 0 2 0 548522235 678875136 14018 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970033 0 0 16781312 514 18446604452685872192 0 0 -1 1 0 0 0
[pid=3062] ppid=2904 vsize=673220 CPUtime=3.06 cores=0-7
/proc/3062/stat : 3062 (python) S 2904 2904 2328 0 -1 4202560 1247 0 0 0 160 146 0 0 18 0 2 0 548512091 689377280 14034 33554432000 4194304 5683644 140733781901008 18446744073709551615 210647166534 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/3062/statm: 168305 14034 407 364 0 104136 0
[pid=3064] ppid=3062 vsize=465044 CPUtime=1244.94 cores=0-7
/proc/3064/stat : 3064 (TNM) R 3062 3064 3064 0 -1 4202496 6038 0 0 0 124475 19 0 0 25 0 1 0 548512092 476205056 5749 33554432000 134512640 135008748 4294106640 18446744073709551615 134528942 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/3064/statm: 116261 5749 46 122 0 115161 0
[pid=3062/tid=3068] ppid=2904 vsize=673220 CPUtime=0 cores=0-7
/proc/3062/task/3068/stat : 3068 (python) S 2904 2904 2328 0 -1 4202560 93 0 0 0 0 0 0 0 19 0 2 0 548522018 689377280 14034 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970033 0 0 16781312 514 18446604445946881280 0 0 -1 1 0 0 0
[pid=3074] ppid=2904 vsize=683476 CPUtime=0.73 cores=0-7
/proc/3074/stat : 3074 (python) S 2904 2904 2328 0 -1 4202560 1273 0 0 0 36 37 0 0 15 0 2 0 548539274 699879424 14072 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658973131 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/3074/statm: 170869 14072 407 364 0 106697 0
[pid=3076] ppid=3074 vsize=268420 CPUtime=296.8 cores=0-7
/proc/3076/stat : 3076 (hybridGM3) T 3074 3076 3076 0 -1 4202496 8115 0 0 0 29675 5 0 0 25 0 1 0 548539275 274862080 7825 33554432000 134512640 135204385 4289479920 18446744073709551615 134537292 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/3076/statm: 67105 7825 64 169 0 66935 0
[pid=3074/tid=3087] ppid=2904 vsize=683476 CPUtime=0 cores=0-7
/proc/3074/task/3087/stat : 3087 (python) S 2904 2904 2328 0 -1 4202560 100 0 0 0 0 0 0 0 15 0 2 0 548558937 699879424 14072 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970033 0 0 16781312 514 18446604451947163712 0 0 -1 3 0 0 0
[pid=3080] ppid=2904 vsize=693732 CPUtime=0.49 cores=0-7
/proc/3080/stat : 3080 (python) S 2904 2904 2328 0 -1 4202560 1262 0 0 0 27 22 0 0 19 0 2 0 548547594 710381568 14082 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658973131 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/3080/statm: 173433 14082 407 364 0 109258 0
[pid=3082] ppid=3080 vsize=431704 CPUtime=194.42 cores=0-7
/proc/3082/stat : 3082 (adaptg2wsat2009) T 3080 3082 3082 0 -1 4202496 6279 0 0 0 19437 5 0 0 25 0 1 0 548547595 442064896 5984 33554432000 134512640 135107411 4290543264 18446744073709551615 134531537 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/3082/statm: 107926 5984 59 146 0 106802 0
[pid=3080/tid=3090] ppid=2904 vsize=693732 CPUtime=0 cores=0-7
/proc/3080/task/3090/stat : 3090 (python) S 2904 2904 2328 0 -1 4202560 94 0 0 0 0 0 0 0 19 0 2 0 548567243 710381568 14082 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970033 0 0 16781312 514 18446604451947164608 0 0 -1 6 0 0 0
[pid=3091] ppid=2904 vsize=703988 CPUtime=1.75 cores=0-7
/proc/3091/stat : 3091 (python) S 2904 2904 2328 0 -1 4202560 1265 0 0 0 96 79 0 0 18 0 2 0 548567254 720883712 14119 33554432000 4194304 5683644 140733781901008 18446744073709551615 210647166534 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/3091/statm: 175997 14119 401 364 0 111819 0
[pid=3093] ppid=3091 vsize=433188 CPUtime=699.14 cores=0-7
/proc/3093/stat : 3093 (NCVWr) R 3091 3093 3093 0 -1 4202496 6446 0 0 0 69911 3 0 0 25 0 1 0 548567255 443584512 6150 33554432000 134512640 135113558 4291082848 18446744073709551615 134531670 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/3093/statm: 108297 6150 58 147 0 107172 0
[pid=3091/tid=3125] ppid=2904 vsize=703988 CPUtime=0 cores=0-7
/proc/3091/task/3125/stat : 3125 (python) S 2904 2904 2328 0 -1 4202560 102 0 0 0 0 0 0 0 19 0 2 0 548616272 720883712 14119 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970033 0 0 16781312 514 18446604445415145600 0 0 -1 4 0 0 0
[pid=3094] ppid=2904 vsize=714244 CPUtime=0.25 cores=0-7
/proc/3094/stat : 3094 (python) S 2904 2904 2328 0 -1 4202560 1256 0 0 0 15 10 0 0 19 0 2 0 548569083 731385856 14187 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658973131 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/3094/statm: 178561 14187 401 364 0 114380 0
[pid=3096] ppid=3094 vsize=260404 CPUtime=99.97 cores=0-7
/proc/3096/stat : 3096 (FH) T 3094 3096 3096 0 -1 4202496 6188 0 0 0 9997 0 0 0 25 0 1 0 548569084 266653696 5898 33554432000 134512640 134936988 4290385568 18446744073709551615 134554295 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/3096/statm: 65101 5898 41 104 0 64605 0
[pid=3094/tid=3101] ppid=2904 vsize=714244 CPUtime=0 cores=0-7
/proc/3094/task/3101/stat : 3101 (python) S 2904 2904 2328 0 -1 4202560 94 0 0 0 0 0 0 0 19 0 2 0 548579108 731385856 14187 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970033 0 0 16781312 514 18446604445415148288 0 0 -1 3 0 0 0
[pid=3128] ppid=2904 vsize=724500 CPUtime=0.44 cores=0-7
/proc/3128/stat : 3128 (python) S 2904 2904 2328 0 -1 4202560 1061 0 0 0 21 23 0 0 15 0 1 0 548619413 741888000 14121 33554432000 4194304 5683644 140733781901008 18446744073709551615 210647166534 0 0 16781312 514 17631539764597326082 0 0 17 0 0 0 0
/proc/3128/statm: 181125 14121 373 364 0 116941 0
[pid=3130] ppid=3128 vsize=268424 CPUtime=178.94 cores=0-7
/proc/3130/stat : 3130 (hybridGM3) R 3128 3130 3130 0 -1 4202496 8112 0 0 0 17893 1 0 0 25 0 1 0 548619414 274866176 7823 33554432000 134512640 135204385 4293788480 18446744073709551615 134537401 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/3130/statm: 67106 7823 64 169 0 66936 0
[pid=3139] ppid=2904 vsize=734756 CPUtime=0 cores=0-7
/proc/3139/stat : 3139 (python) S 2904 2904 2328 0 -1 4202560 944 0 0 0 0 0 0 0 19 0 1 0 548637361 752390144 14032 33554432000 4194304 5683644 140733781901008 18446744073709551615 210647166534 0 0 16781312 514 17537201396351238143 0 0 17 0 0 0 0
/proc/3139/statm: 183689 14032 346 364 0 119502 0
[pid=3141] ppid=3139 vsize=412620 CPUtime=0.01 cores=0-7
/proc/3141/stat : 3141 (NCVWr) R 3139 3141 3141 0 -1 4202496 628 0 0 0 1 0 0 0 21 0 1 0 548637362 422522880 338 33554432000 134512640 135113558 4293613744 18446744073709551615 134619510 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/3141/statm: 103155 338 55 147 0 102030 0
[pid=2904/tid=2912] ppid=2902 vsize=745000 CPUtime=620.36 cores=0-7
/proc/2904/task/2912/stat : 2912 (python) S 2902 2904 2328 0 -1 4202560 39 19584 0 0 0 0 61932 104 18 0 37 0 548403268 762880000 15629 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970033 0 0 16781312 2 18446604442507624512 0 0 -1 6 0 0 0
[pid=2904/tid=2915] ppid=2902 vsize=745000 CPUtime=620.36 cores=0-7
/proc/2904/task/2915/stat : 2915 (python) S 2902 2904 2328 0 -1 4202560 43 19584 0 0 0 0 61932 104 18 0 37 0 548403272 762880000 15629 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970033 0 0 16781312 2 18446604442507624512 0 0 -1 2 0 0 0
[pid=2904/tid=2918] ppid=2902 vsize=745000 CPUtime=620.36 cores=0-7
/proc/2904/task/2918/stat : 2918 (python) S 2902 2904 2328 0 -1 4202560 53 19584 0 0 0 0 61932 104 15 0 37 0 548403276 762880000 15629 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970033 0 0 16781312 2 18446604442507624512 0 0 -1 6 0 0 0
[pid=2904/tid=2921] ppid=2902 vsize=745000 CPUtime=620.36 cores=0-7
/proc/2904/task/2921/stat : 2921 (python) S 2902 2904 2328 0 -1 4202560 46 19584 0 0 0 0 61932 104 18 0 37 0 548403280 762880000 15629 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970033 0 0 16781312 2 18446604442507624512 0 0 -1 6 0 0 0
[pid=2904/tid=2924] ppid=2902 vsize=745000 CPUtime=620.36 cores=0-7
/proc/2904/task/2924/stat : 2924 (python) S 2902 2904 2328 0 -1 4202560 42 19584 0 0 0 0 61932 104 15 0 37 0 548403284 762880000 15629 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970033 0 0 16781312 2 18446604442507624512 0 0 -1 7 0 0 0
[pid=2904/tid=2927] ppid=2902 vsize=745000 CPUtime=620.36 cores=0-7
/proc/2904/task/2927/stat : 2927 (python) S 2902 2904 2328 0 -1 4202560 118 19584 0 0 0 0 61932 104 15 0 37 0 548403289 762880000 15629 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970033 0 0 16781312 2 18446604442507624512 0 0 -1 1 0 0 0
[pid=2904/tid=2932] ppid=2902 vsize=745000 CPUtime=620.36 cores=0-7
/proc/2904/task/2932/stat : 2932 (python) S 2902 2904 2328 0 -1 4202560 40 19584 0 0 0 0 61932 104 18 0 37 0 548403293 762880000 15629 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970033 0 0 16781312 2 18446604442507624512 0 0 -1 3 0 0 0
[pid=2904/tid=2934] ppid=2902 vsize=745000 CPUtime=620.36 cores=0-7
/proc/2904/task/2934/stat : 2934 (python) S 2902 2904 2328 0 -1 4202560 94 19584 0 0 0 0 61932 104 18 0 37 0 548403297 762880000 15629 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970033 0 0 16781312 2 18446604442507624512 0 0 -1 0 0 0 0
[pid=2904/tid=2944] ppid=2902 vsize=745000 CPUtime=620.36 cores=0-7
/proc/2904/task/2944/stat : 2944 (python) S 2902 2904 2328 0 -1 4202560 48 19584 0 0 0 0 61932 104 18 0 37 0 548413325 762880000 15629 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970033 0 0 16781312 2 18446604442507624512 0 0 -1 2 0 0 0
[pid=2904/tid=2959] ppid=2902 vsize=745000 CPUtime=620.36 cores=0-7
/proc/2904/task/2959/stat : 2959 (python) S 2902 2904 2328 0 -1 4202560 64 19584 0 0 0 0 61932 104 18 0 37 0 548432453 762880000 15629 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970033 0 0 16781312 2 18446604442507624512 0 0 -1 1 0 0 0
[pid=2904/tid=2963] ppid=2902 vsize=745000 CPUtime=620.36 cores=0-7
/proc/2904/task/2963/stat : 2963 (python) S 2902 2904 2328 0 -1 4202560 41 19584 0 0 0 0 61932 104 18 0 37 0 548432465 762880000 15629 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970033 0 0 16781312 2 18446604442507624512 0 0 -1 0 0 0 0
[pid=2904/tid=2966] ppid=2902 vsize=745000 CPUtime=620.36 cores=0-7
/proc/2904/task/2966/stat : 2966 (python) S 2902 2904 2328 0 -1 4202560 50 19584 0 0 0 0 61932 104 18 0 37 0 548432503 762880000 15629 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970033 0 0 16781312 2 18446604442507624512 0 0 -1 0 0 0 0
[pid=2904/tid=2969] ppid=2902 vsize=745000 CPUtime=620.36 cores=0-7
/proc/2904/task/2969/stat : 2969 (python) S 2902 2904 2328 0 -1 4202560 38 19584 0 0 0 0 61932 104 18 0 37 0 548432761 762880000 15629 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970033 0 0 16781312 2 18446604442507624512 0 0 -1 0 0 0 0
[pid=2904/tid=2977] ppid=2902 vsize=745000 CPUtime=620.36 cores=0-7
/proc/2904/task/2977/stat : 2977 (python) S 2902 2904 2328 0 -1 4202560 42 19584 0 0 0 0 61932 104 20 0 37 0 548442802 762880000 15629 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970033 0 0 16781312 2 18446604442507624512 0 0 -1 0 0 0 0
[pid=2904/tid=2981] ppid=2902 vsize=745000 CPUtime=620.36 cores=0-7
/proc/2904/task/2981/stat : 2981 (python) S 2902 2904 2328 0 -1 4202560 46 19584 0 0 0 0 61932 104 18 0 37 0 548449011 762880000 15629 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970033 0 0 16781312 2 18446604442507624512 0 0 -1 1 0 0 0
[pid=2904/tid=2992] ppid=2902 vsize=745000 CPUtime=620.36 cores=0-7
/proc/2904/task/2992/stat : 2992 (python) S 2902 2904 2328 0 -1 4202560 121 19584 0 0 0 0 61932 104 18 0 37 0 548459039 762880000 15629 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970033 0 0 16781312 2 18446604442507624512 0 0 -1 5 0 0 0
[pid=2904/tid=2995] ppid=2902 vsize=745000 CPUtime=620.36 cores=0-7
/proc/2904/task/2995/stat : 2995 (python) S 2902 2904 2328 0 -1 4202560 39 19584 0 0 0 0 61932 104 18 0 37 0 548459048 762880000 15629 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970033 0 0 16781312 2 18446604442507624512 0 0 -1 0 0 0 0
[pid=2904/tid=2999] ppid=2902 vsize=745000 CPUtime=620.36 cores=0-7
/proc/2904/task/2999/stat : 2999 (python) S 2902 2904 2328 0 -1 4202560 39 19584 0 0 0 0 61932 104 20 0 37 0 548462276 762880000 15629 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970033 0 0 16781312 2 18446604442507624512 0 0 -1 0 0 0 0
[pid=2904/tid=3002] ppid=2902 vsize=745000 CPUtime=620.36 cores=0-7
/proc/2904/task/3002/stat : 3002 (python) S 2902 2904 2328 0 -1 4202560 33 19584 0 0 0 0 61932 104 21 0 37 0 548462290 762880000 15629 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970033 0 0 16781312 2 18446604442507624512 0 0 -1 0 0 0 0
[pid=2904/tid=3005] ppid=2902 vsize=745000 CPUtime=620.36 cores=0-7
/proc/2904/task/3005/stat : 3005 (python) S 2902 2904 2328 0 -1 4202560 48 19584 0 0 0 0 61932 104 18 0 37 0 548462757 762880000 15629 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970033 0 0 16781312 2 18446604442507624512 0 0 -1 0 0 0 0
[pid=2904/tid=3009] ppid=2902 vsize=745000 CPUtime=620.36 cores=0-7
/proc/2904/task/3009/stat : 3009 (python) S 2902 2904 2328 0 -1 4202560 36 19584 0 0 0 0 61932 104 20 0 37 0 548462986 762880000 15629 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970033 0 0 16781312 2 18446604442507624512 0 0 -1 0 0 0 0
[pid=2904/tid=3014] ppid=2902 vsize=745000 CPUtime=620.36 cores=0-7
/proc/2904/task/3014/stat : 3014 (python) S 2902 2904 2328 0 -1 4202560 35 19584 0 0 0 0 61932 104 20 0 37 0 548471769 762880000 15629 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970033 0 0 16781312 2 18446604442507624512 0 0 -1 1 0 0 0
[pid=2904/tid=3018] ppid=2902 vsize=745000 CPUtime=620.36 cores=0-7
/proc/2904/task/3018/stat : 3018 (python) S 2902 2904 2328 0 -1 4202560 32 19584 0 0 0 0 61932 104 19 0 37 0 548472207 762880000 15629 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970033 0 0 16781312 2 18446604442507624512 0 0 -1 0 0 0 0
[pid=2904/tid=3024] ppid=2902 vsize=745000 CPUtime=620.36 cores=0-7
/proc/2904/task/3024/stat : 3024 (python) S 2902 2904 2328 0 -1 4202560 44 19584 0 0 0 0 61932 104 18 0 37 0 548472919 762880000 15629 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970033 0 0 16781312 2 18446604442507624512 0 0 -1 0 0 0 0
[pid=2904/tid=3032] ppid=2902 vsize=745000 CPUtime=620.36 cores=0-7
/proc/2904/task/3032/stat : 3032 (python) S 2902 2904 2328 0 -1 4202560 43 19584 0 0 0 0 61932 104 18 0 37 0 548481801 762880000 15629 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970033 0 0 16781312 2 18446604442507624512 0 0 -1 0 0 0 0
[pid=2904/tid=3036] ppid=2902 vsize=745000 CPUtime=620.36 cores=0-7
/proc/2904/task/3036/stat : 3036 (python) S 2902 2904 2328 0 -1 4202560 38 19584 0 0 0 0 61932 104 18 0 37 0 548482240 762880000 15629 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970033 0 0 16781312 2 18446604442507624512 0 0 -1 0 0 0 0
[pid=2904/tid=3042] ppid=2902 vsize=745000 CPUtime=620.36 cores=0-7
/proc/2904/task/3042/stat : 3042 (python) S 2902 2904 2328 0 -1 4202560 52 19584 0 0 0 0 61932 104 18 0 37 0 548488842 762880000 15629 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970033 0 0 16781312 2 18446604442507624512 0 0 -1 0 0 0 0
[pid=2904/tid=3046] ppid=2902 vsize=745000 CPUtime=620.36 cores=0-7
/proc/2904/task/3046/stat : 3046 (python) S 2902 2904 2328 0 -1 4202560 42 19584 0 0 0 0 61932 104 18 0 37 0 548492557 762880000 15629 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970033 0 0 16781312 2 18446604442507624512 0 0 -1 0 0 0 0
[pid=2904/tid=3056] ppid=2902 vsize=745000 CPUtime=620.36 cores=0-7
/proc/2904/task/3056/stat : 3056 (python) S 2902 2904 2328 0 -1 4202560 36 19584 0 0 0 0 61932 104 19 0 37 0 548502590 762880000 15629 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970033 0 0 16781312 2 18446604442507624512 0 0 -1 0 0 0 0
[pid=2904/tid=3063] ppid=2902 vsize=745000 CPUtime=620.36 cores=0-7
/proc/2904/task/3063/stat : 3063 (python) S 2902 2904 2328 0 -1 4202560 33 19584 0 0 0 0 61932 104 18 0 37 0 548512092 762880000 15629 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970033 0 0 16781312 2 18446604442507624512 0 0 -1 1 0 0 0
[pid=2904/tid=3075] ppid=2902 vsize=745000 CPUtime=620.36 cores=0-7
/proc/2904/task/3075/stat : 3075 (python) S 2902 2904 2328 0 -1 4202560 40 19584 0 0 0 0 61932 104 18 0 37 0 548539275 762880000 15629 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970033 0 0 16781312 2 18446604442507624512 0 0 -1 0 0 0 0
[pid=2904/tid=3081] ppid=2902 vsize=745000 CPUtime=620.36 cores=0-7
/proc/2904/task/3081/stat : 3081 (python) S 2902 2904 2328 0 -1 4202560 36 19584 0 0 0 0 61932 104 19 0 37 0 548547594 762880000 15629 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970033 0 0 16781312 2 18446604442507624512 0 0 -1 0 0 0 0
[pid=2904/tid=3092] ppid=2902 vsize=745000 CPUtime=620.36 cores=0-7
/proc/2904/task/3092/stat : 3092 (python) S 2902 2904 2328 0 -1 4202560 43 19584 0 0 0 0 61932 104 18 0 37 0 548567254 762880000 15629 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970033 0 0 16781312 2 18446604442507624512 0 0 -1 0 0 0 0
[pid=2904/tid=3095] ppid=2902 vsize=745000 CPUtime=620.36 cores=0-7
/proc/2904/task/3095/stat : 3095 (python) S 2902 2904 2328 0 -1 4202560 35 19584 0 0 0 0 61932 104 20 0 37 0 548569083 762880000 15629 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970033 0 0 16781312 2 18446604442507624512 0 0 -1 0 0 0 0
[pid=2904/tid=3129] ppid=2902 vsize=745000 CPUtime=620.36 cores=0-7
/proc/2904/task/3129/stat : 3129 (python) S 2902 2904 2328 0 -1 4202560 38 19584 0 0 0 0 61932 104 20 0 37 0 548619413 762880000 15629 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970033 0 0 16781312 2 18446604442507624512 0 0 -1 0 0 0 0
[pid=2904/tid=3140] ppid=2902 vsize=745000 CPUtime=620.36 cores=0-7
/proc/2904/task/3140/stat : 3140 (python) S 2902 2904 2328 0 -1 4202560 37 19584 0 0 0 0 61932 104 20 0 37 0 548637362 762880000 15629 33554432000 4194304 5683644 140733781901008 18446744073709551615 210658970033 0 0 16781312 2 18446604442507624512 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 18619.2
Current children cumulated vsize (KiB) 31789792

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 2904 and gives
#  childrusage.ru_utime.tv_sec=633
#  childrusage.ru_utime.tv_usec=856639
#  childrusage.ru_stime.tv_sec=1
#  childrusage.ru_stime.tv_usec=428782
# CPU time returned by wait4() is 635.285
# while last known CPU time is 18619.2
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 2348.62
CPU time (s): 18619.2
CPU user time (s): 18594.1
CPU system time (s): 25.08
CPU usage (%): 792.774
Max. virtual memory (cumulated for all children) (KiB): 31789792

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 633.857
system time used= 1.42878
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 128093
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 3990
involuntary context switches= 5483

runsolver used 23.7604 second user time and 63.5913 second system time

The end

Launcher Data 

Begin job on node129 at 2011-05-02 22:27:09
IDJOB=3321832
IDBENCH=82610
IDSOLVER=1712
FILE ID=node129/3321832-1304368028
RUNJOBID= node129-1304366466-2342
PBS_JOBID= 13181742
Free space on /tmp= 67572 MiB

SOLVER NAME= borg-sat 11.03.02-random
BENCH NAME= SAT11/random/large/unif-k3-r4.2-v35000-c147000-S970100151-015-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3321832-1304368028/watcher-3321832-1304368028 -o /tmp/evaluation-result-3321832-1304368028/solver-3321832-1304368028 -C 40000 -W 5100 -M 31000  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3321832-1304368028.cnf -seed 1174201737 -budget 40000 -cores 8

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 1ba3e47e885ea3862e5726ce64a96a9c
RANDOM SEED=1174201737

node129.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951132 kB
MemFree:      27634196 kB
Buffers:         58184 kB
Cached:        1110424 kB
SwapCached:      38596 kB
Active:        4242560 kB
Inactive:       899956 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951132 kB
LowFree:      27634196 kB
SwapTotal:    67111528 kB
SwapFree:     65145212 kB
Dirty:           34604 kB
Writeback:           0 kB
AnonPages:     3945092 kB
Mapped:          11632 kB
Slab:            82496 kB
PageTables:      25844 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587092 kB
Committed_AS: 45784140 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296000 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 67476 MiB
End job on node129 at 2011-05-02 23:06:20