Trace number 3321730

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-random? (MO) 18933.6 2384.92

General information on the benchmark

Namerandom/large/
unif-k3-r4.2-v35000-c147000-S883457431-033-UNKNOWN.cnf
MD5SUM0d0abbf84f73b4226aa3bedf08367d2e
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark132.892
Satisfiable
(Un)Satisfiability was proved
Number of variables35000
Number of clauses147000
Sum of the clauses size441000
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 3147000
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.35/0.67	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.35/0.67	c INFO: solving HOME/instance-3321730-1304366507.cnf
0.35/0.67	c INFO: computing task features
0.84/1.17	c INFO: computed variable-clause graph statistics
0.84/1.20	c INFO: computed clause balance statistics
3.53/3.80	c INFO: computed variable balance statistics
5.63/5.90	c INFO: computed small-clause counts
5.63/5.92	c INFO: computed Horn-clause counts
7.12/7.47	c INFO: computed variable graph statistics
7.12/7.47	c INFO: collected features for HOME/instance-3321730-1304366507.cnf in 6.78 s
7.22/7.51	c INFO: running hybridGM3@0 for 97 with 39993 remaining (b = 0.36)
7.22/7.55	c INFO: running hybridGM3@0 for 97 with 39993 remaining (b = 0.21)
7.22/7.59	c INFO: running hybridGM3@0 for 97 with 39993 remaining (b = 0.14)
7.43/7.63	c INFO: running hybridGM3@0 for 194 with 39993 remaining (b = 0.18)
7.43/7.67	c INFO: running hybridGM3@0 for 194 with 39993 remaining (b = 0.14)
7.84/7.71	c INFO: running hybridGM3@0 for 194 with 39992 remaining (b = 0.11)
7.84/7.75	c INFO: running TNM@0 for 97 with 39992 remaining (b = 0.04)
7.84/7.79	c INFO: running hybridGM3@0 for 194 with 39992 remaining (b = 0.07)
806.96/107.79	c INFO: running hybridGM3@0 for 97 with 39892 remaining (b = 0.03)
807.76/107.83	c INFO: running hybridGM3@0 for 97 with 39892 remaining (b = 0.03)
807.76/107.87	c INFO: running hybridGM3@0 for 97 with 39892 remaining (b = 0.03)
808.49/107.96	c INFO: running hybridGM3@0 for 194 with 39892 remaining (b = 0.06)
1566.09/202.96	c INFO: running hybridGM3@100 for 97 with 39797 remaining (b = 0.03)
1566.88/203.00	c INFO: running hybridGM3@100 for 97 with 39797 remaining (b = 0.03)
1566.88/203.05	c INFO: running hybridGM3@99 for 97 with 39797 remaining (b = 0.03)
1574.85/204.08	c INFO: running hybridGM3@0 for 97 with 39796 remaining (b = 0.03)
1575.62/204.13	c INFO: running hybridGM3@0 for 97 with 39796 remaining (b = 0.03)
1576.40/204.21	c INFO: running hybridGM3@100 for 97 with 39796 remaining (b = 0.03)
1622.74/210.04	c INFO: running hybridGM3@0 for 97 with 39790 remaining (b = 0.03)
2334.87/299.25	c INFO: running hybridGM3@0 for 194 with 39701 remaining (b = 0.04)
2373.91/304.14	c INFO: running TNM@0 for 97 with 39696 remaining (b = 0.02)
2374.65/304.20	c INFO: running hybridGM3@0 for 194 with 39696 remaining (b = 0.03)
2374.65/304.25	c INFO: running hybridGM3@295 for 97 with 39696 remaining (b = 0.02)
2375.84/304.43	c INFO: running hybridGM3@94 for 97 with 39696 remaining (b = 0.02)
2382.53/305.22	c INFO: running hybridGM3@94 for 97 with 39695 remaining (b = 0.02)
2383.28/305.31	c INFO: running TNM@0 for 1940 with 39695 remaining (b = 0.22)
2384.07/305.40	c INFO: running hybridGM3@300 for 97 with 39695 remaining (b = 0.02)
3134.50/399.44	c INFO: running hybridGM3@0 for 194 with 39601 remaining (b = 0.03)
3141.69/400.40	c INFO: running TNM@0 for 97 with 39600 remaining (b = 0.01)
3143.12/400.59	c INFO: running hybridGM3@394 for 97 with 39600 remaining (b = 0.01)
3173.48/404.38	c INFO: running hybridGM3@0 for 194 with 39596 remaining (b = 0.02)
3183.84/405.63	c INFO: running hybridGM3@189 for 97 with 39595 remaining (b = 0.01)
3902.87/495.75	c INFO: running hybridGM3@100 for 97 with 39504 remaining (b = 0.01)
3903.64/495.81	c INFO: running TNM@0 for 97 with 39504 remaining (b = 0.01)
3941.90/500.61	c INFO: running adaptg2wsat2009++@0 for 194 with 39500 remaining (b = 0.02)
3943.31/500.83	c INFO: running hybridGM3@385 for 97 with 39499 remaining (b = 0.01)
3990.50/506.77	c INFO: running hybridGM3@101 for 97 with 39493 remaining (b = 0.01)
4694.59/595.03	c INFO: running TNM@0 for 97 with 39405 remaining (b = 0.01)
4700.87/595.96	c INFO: running hybridGM3@0 for 194 with 39404 remaining (b = 0.02)
4701.50/596.04	c INFO: running hybridGM3@95 for 97 with 39404 remaining (b = 0.01)
4741.48/600.96	c INFO: running hybridGM3@100 for 97 with 39399 remaining (b = 0.01)
4749.44/601.99	c INFO: running TNM@0 for 97 with 39398 remaining (b = 0.01)
4750.23/602.07	c INFO: running hybridGM3@0 for 194 with 39398 remaining (b = 0.01)
5461.40/691.26	c INFO: running hybridGM3@291 for 194 with 39309 remaining (b = 0.02)
5485.31/694.28	c INFO: running TNM@0 for 97 with 39306 remaining (b = 0.01)
5501.28/696.22	c INFO: running TNM@0 for 97 with 39304 remaining (b = 0.01)
5506.19/696.99	c INFO: running TNM@0 for 97 with 39303 remaining (b = 0.01)
5588.83/707.27	c INFO: running adaptg2wsat2009++@94 for 97 with 39293 remaining (b = 0.01)
6268.10/792.52	c INFO: running adaptg2wsat2009++@0 for 194 with 39208 remaining (b = 0.01)
6276.83/793.62	c INFO: running TNM@0 for 97 with 39207 remaining (b = 0.01)
6283.98/794.51	c INFO: running TNM@195 for 1843 with 39206 remaining (b = 0.12)
6316.71/798.62	c INFO: running hybridGM3@94 for 97 with 39202 remaining (b = 0.01)
6353.26/803.29	c INFO: running hybridGM3@0 for 194 with 39197 remaining (b = 0.01)
6355.42/803.54	c INFO: running adaptg2wsat2009++@0 for 194 with 39197 remaining (b = 0.01)
7033.26/888.66	c INFO: running hybridGM3@390 for 97 with 39112 remaining (b = 0.01)
7074.69/893.89	c INFO: running hybridGM3@195 for 291 with 39106 remaining (b = 0.02)
7122.49/899.88	c INFO: running FH@0 for 97 with 39100 remaining (b = 0.01)
7792.01/983.92	c INFO: running hybridGM3@194 for 291 with 39016 remaining (b = 0.02)
7817.50/987.14	c INFO: running adaptg2wsat2009++@99 for 97 with 39013 remaining (b = 0.01)
7918.64/999.85	c INFO: running hybridGM3@95 for 97 with 39000 remaining (b = 0.01)
7920.97/1000.17	c INFO: running hybridGM3@195 for 291 with 39000 remaining (b = 0.02)
7952.01/1004.00	c INFO: running hybridGM3@196 for 291 with 38996 remaining (b = 0.01)
8663.64/1093.43	c INFO: running adaptg2wsat2009++@295 for 291 with 38907 remaining (b = 0.02)
8677.12/1095.11	c INFO: running TNM@0 for 97 with 38905 remaining (b = 0.00)
9427.14/1189.49	c INFO: running hybridGM3@293 for 194 with 38811 remaining (b = 0.01)
9474.03/1195.40	c INFO: running hybridGM3@196 for 291 with 38805 remaining (b = 0.01)
10096.03/1273.52	c INFO: running hybridGM3@389 for 97 with 38727 remaining (b = 0.01)
10216.90/1288.76	c INFO: running hybridGM3@391 for 97 with 38712 remaining (b = 0.01)
10255.12/1293.59	c INFO: running hybridGM3@298 for 194 with 38707 remaining (b = 0.01)
10900.93/1374.80	c INFO: running TNM@295 for 1746 with 38625 remaining (b = 0.09)
10950.22/1380.92	c INFO: running hybridGM3@393 for 97 with 38619 remaining (b = 0.01)
11022.57/1390.04	c INFO: running hybridGM3@195 for 291 with 38610 remaining (b = 0.01)
11071.11/1396.17	c INFO: running adaptg2wsat2009++@390 for 194 with 38604 remaining (b = 0.01)
11707.39/1476.19	c INFO: running hybridGM3@199 for 291 with 38524 remaining (b = 0.01)
11777.40/1485.00	c INFO: running hybridGM3@484 for 97 with 38515 remaining (b = 0.01)
11778.82/1485.11	c INFO: running hybridGM3@398 for 97 with 38515 remaining (b = 0.01)
12583.02/1586.27	c INFO: running hybridGM3@0 for 194 with 38414 remaining (b = 0.01)
12583.72/1586.39	c INFO: running NCVWr@0 for 485 with 38414 remaining (b = 0.02)
12689.50/1599.63	c INFO: running adaptg2wsat2009++@485 for 97 with 38401 remaining (b = 0.01)
13325.82/1679.65	c INFO: running hybridGM3@483 for 97 with 38321 remaining (b = 0.01)
13495.17/1700.92	c INFO: running adaptg2wsat2009++@774 for 97 with 38299 remaining (b = 0.01)
14059.07/1771.81	c INFO: running FH@0 for 97 with 38228 remaining (b = 0.00)
14083.72/1774.95	c INFO: running hybridGM3@195 for 291 with 38225 remaining (b = 0.01)
14146.56/1782.87	c INFO: running hybridGM3@100 for 97 with 38217 remaining (b = 0.00)
14300.84/1802.21	c INFO: running hybridGM3@196 for 291 with 38198 remaining (b = 0.01)
14848.87/1871.14	c INFO: running hybridGM3@197 for 291 with 38129 remaining (b = 0.01)
14952.23/1884.16	c INFO: running TNM@100 for 1455 with 38116 remaining (b = 0.07)
16434.70/2070.57	c INFO: running hybridGM3@391 for 97 with 37930 remaining (b = 0.01)
16561.16/2086.47	c INFO: running NCVWr@193 for 388 with 37914 remaining (b = 0.02)
16604.08/2091.81	c INFO: running hybridGM3@295 for 194 with 37908 remaining (b = 0.01)
17192.66/2165.85	c INFO: running hybridGM3@194 for 291 with 37834 remaining (b = 0.01)
17199.79/2166.76	c INFO: running hybridGM3@298 for 194 with 37834 remaining (b = 0.01)
17838.39/2247.02	c INFO: running TNM@94 for 1358 with 37753 remaining (b = 0.06)
18174.01/2289.26	c INFO: running hybridGM3@390 for 97 with 37711 remaining (b = 0.01)
18722.03/2358.19	c INFO: running hybridGM3@0 for 194 with 37642 remaining (b = 0.01)
18931.96/2384.56	c INFO: running NCVWr@0 for 485 with 37616 remaining (b = 0.02)
18933.63/2384.90	c INFO: running FH@0 for 97 with 37615 remaining (b = 0.00)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3321730-1304366507/watcher-3321730-1304366507 -o /tmp/evaluation-result-3321730-1304366507/solver-3321730-1304366507 -C 40000 -W 5100 -M 31000 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3321730-1304366507.cnf -seed 1397071266 -budget 40000 -cores 8 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 1.75 0.44 0.15 2/191 1316
/proc/meminfo: memFree=18067524/32950904 swapFree=67082284/67111528
[pid=1316] ppid=1314 vsize=15080 CPUtime=0 cores=0-7
/proc/1316/stat : 1316 (python) S 1314 1316 1221 0 -1 4202496 616 0 0 0 0 0 0 0 20 0 1 0 548250520 15441920 512 33554432000 4194304 5683644 140733331201824 18446744073709551615 237278877696 0 2147483391 16781312 2 18446744071699199166 0 0 17 6 0 0 0
/proc/1316/statm: 3770 512 300 364 0 257 0

[startup+0.0169329 s]
/proc/loadavg: 1.75 0.44 0.15 2/191 1316
/proc/meminfo: memFree=18067524/32950904 swapFree=67082284/67111528
[pid=1316] ppid=1314 vsize=16236 CPUtime=0 cores=0-7
/proc/1316/stat : 1316 (python) R 1314 1316 1221 0 -1 4202496 997 0 0 0 0 0 0 0 20 0 1 0 548250520 16625664 869 33554432000 4194304 5683644 140733331201824 18446744073709551615 4426144 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/1316/statm: 4059 869 368 364 0 546 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 16236

[startup+0.101011 s]
/proc/loadavg: 1.75 0.44 0.15 2/191 1316
/proc/meminfo: memFree=18067524/32950904 swapFree=67082284/67111528
[pid=1316] ppid=1314 vsize=89764 CPUtime=0.06 cores=0-7
/proc/1316/stat : 1316 (python) R 1314 1316 1221 0 -1 4202496 2672 0 0 0 4 2 0 0 20 0 1 0 548250520 91918336 2387 33554432000 4194304 5683644 140733331201824 18446744073709551615 237278877696 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/1316/statm: 22441 2387 750 364 0 1621 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 89764

[startup+0.300993 s]
/proc/loadavg: 1.75 0.44 0.15 2/191 1316
/proc/meminfo: memFree=18067524/32950904 swapFree=67082284/67111528
[pid=1316] ppid=1314 vsize=135748 CPUtime=0.19 cores=0-7
/proc/1316/stat : 1316 (python) R 1314 1316 1221 0 -1 4202496 5964 934 1 0 13 6 0 0 20 0 1 0 548250520 139005952 4151 33554432000 4194304 5683644 140733331201824 18446744073709551615 237278875653 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/1316/statm: 33937 4151 1032 364 0 3123 0
Current children cumulated CPU time (s) 0.19
Current children cumulated vsize (KiB) 135748

[startup+0.700895 s]
/proc/loadavg: 1.75 0.44 0.15 2/191 1316
/proc/meminfo: memFree=18067524/32950904 swapFree=67082284/67111528
[pid=1316] ppid=1314 vsize=282800 CPUtime=0.43 cores=0-7
/proc/1316/stat : 1316 (python) R 1314 1316 1221 0 -1 4202496 10340 934 1 0 31 12 0 0 19 0 1 0 548250520 289587200 8021 33554432000 4194304 5683644 140733331201824 18446744073709551615 4539731 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/1316/statm: 70700 8021 1761 364 0 6624 0
Current children cumulated CPU time (s) 0.43
Current children cumulated vsize (KiB) 282800

[startup+1.50067 s]
/proc/loadavg: 1.75 0.44 0.15 2/192 1321
/proc/meminfo: memFree=18022432/32950904 swapFree=67082284/67111528
[pid=1316] ppid=1314 vsize=312972 CPUtime=1.22 cores=0-7
/proc/1316/stat : 1316 (python) R 1314 1316 1221 0 -1 4202496 21968 934 1 0 109 13 0 0 20 0 1 0 548250520 320483328 15628 33554432000 4194304 5683644 140733331201824 18446744073709551615 4500664 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/1316/statm: 78243 15628 1826 364 0 14167 0
Current children cumulated CPU time (s) 1.22
Current children cumulated vsize (KiB) 312972

[startup+3.10038 s]
/proc/loadavg: 1.75 0.44 0.15 2/192 1321
/proc/meminfo: memFree=18011148/32950904 swapFree=67082284/67111528
[pid=1316] ppid=1314 vsize=312972 CPUtime=2.82 cores=0-7
/proc/1316/stat : 1316 (python) R 1314 1316 1221 0 -1 4202496 21968 934 1 0 269 13 0 0 25 0 1 0 548250520 320483328 15628 33554432000 4194304 5683644 140733331201824 18446744073709551615 47492733120128 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/1316/statm: 78243 15628 1826 364 0 14167 0
Current children cumulated CPU time (s) 2.82
Current children cumulated vsize (KiB) 312972

[startup+6.30081 s]
/proc/loadavg: 1.69 0.45 0.15 2/192 1321
/proc/meminfo: memFree=18011912/32950904 swapFree=67082284/67111528
[pid=1316] ppid=1314 vsize=312972 CPUtime=6.03 cores=0-7
/proc/1316/stat : 1316 (python) R 1314 1316 1221 0 -1 4202496 21975 934 1 0 590 13 0 0 25 0 1 0 548250520 320483328 15635 33554432000 4194304 5683644 140733331201824 18446744073709551615 47492733040780 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/1316/statm: 78243 15635 1833 364 0 14167 0
Current children cumulated CPU time (s) 6.03
Current children cumulated vsize (KiB) 312972

[startup+12.7017 s]
/proc/loadavg: 1.79 0.49 0.17 9/216 1346
/proc/meminfo: memFree=17734720/32950904 swapFree=67082284/67111528
[pid=1316] ppid=1314 vsize=392300 CPUtime=7.52 cores=0-7
/proc/1316/stat : 1316 (python) S 1314 1316 1221 0 -1 4202496 28808 934 1 0 736 16 0 0 25 0 9 0 548250520 401715200 15040 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290706379 0 0 16781312 2 18446744073709551615 0 0 17 0 0 0 0
/proc/1316/statm: 98075 15040 1889 364 0 33972 0
[pid=1322] ppid=1316 vsize=310264 CPUtime=0 cores=0-7
/proc/1322/stat : 1322 (python) S 1316 1316 1221 0 -1 4202560 908 0 0 0 0 0 0 0 15 0 1 0 548251272 317710336 13495 33554432000 4194304 5683644 140733331201824 18446744073709551615 237278899727 0 0 16781312 514 4295320243527450881 0 0 17 1 0 0 0
/proc/1322/statm: 77566 13495 376 364 0 13484 0
[pid=1324] ppid=1322 vsize=268428 CPUtime=5.16 cores=0-7
/proc/1324/stat : 1324 (hybridGM3) R 1322 1324 1324 0 -1 4202496 8082 0 0 0 516 0 0 0 25 0 1 0 548251272 274870272 7819 33554432000 134512640 135204385 4286659248 18446744073709551615 134545975 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/1324/statm: 67107 7819 64 169 0 66937 0
[pid=1325] ppid=1316 vsize=320520 CPUtime=0 cores=0-7
/proc/1325/stat : 1325 (python) S 1316 1316 1221 0 -1 4202560 910 0 0 0 0 0 0 0 15 0 1 0 548251276 328212480 13493 33554432000 4194304 5683644 140733331201824 18446744073709551615 237278899782 0 0 16781312 514 4253170190398750980 0 0 17 4 0 0 0
/proc/1325/statm: 80130 13493 370 364 0 16045 0
[pid=1327] ppid=1325 vsize=268424 CPUtime=5.13 cores=0-7
/proc/1327/stat : 1327 (hybridGM3) R 1325 1327 1327 0 -1 4202496 8079 0 0 0 512 1 0 0 25 0 1 0 548251276 274866176 7818 33554432000 134512640 135204385 4289545936 18446744073709551615 134536821 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/1327/statm: 67106 7818 64 169 0 66936 0
[pid=1328] ppid=1316 vsize=330776 CPUtime=0 cores=0-7
/proc/1328/stat : 1328 (python) S 1316 1316 1221 0 -1 4202560 909 0 0 0 0 0 0 0 15 0 1 0 548251280 338714624 13497 33554432000 4194304 5683644 140733331201824 18446744073709551615 237278899782 0 0 16781312 514 4274339087768322308 0 0 17 6 0 0 0
/proc/1328/statm: 82694 13497 370 364 0 18606 0
[pid=1330] ppid=1328 vsize=268428 CPUtime=5.08 cores=0-7
/proc/1330/stat : 1330 (hybridGM3) R 1328 1330 1330 0 -1 4202496 8087 0 0 0 507 1 0 0 25 0 1 0 548251280 274870272 7823 33554432000 134512640 135204385 4293657904 18446744073709551615 134537049 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/1330/statm: 67107 7823 64 169 0 66937 0
[pid=1331] ppid=1316 vsize=341032 CPUtime=0 cores=0-7
/proc/1331/stat : 1331 (python) S 1316 1316 1221 0 -1 4202560 939 0 0 0 0 0 0 0 15 0 1 0 548251283 349216768 13501 33554432000 4194304 5683644 140733331201824 18446744073709551615 237278899782 0 0 16781312 514 4392090735789179137 0 0 17 4 0 0 0
/proc/1331/statm: 85258 13501 370 364 0 21167 0
[pid=1333] ppid=1331 vsize=268424 CPUtime=5.02 cores=0-7
/proc/1333/stat : 1333 (hybridGM3) R 1331 1333 1333 0 -1 4202496 8084 0 0 0 501 1 0 0 25 0 1 0 548251284 274866176 7822 33554432000 134512640 135204385 4294121376 18446744073709551615 134546059 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/1333/statm: 67106 7822 64 169 0 66936 0
[pid=1334] ppid=1316 vsize=351288 CPUtime=0 cores=0-7
/proc/1334/stat : 1334 (python) S 1316 1316 1221 0 -1 4202560 944 0 0 0 0 0 0 0 15 0 1 0 548251287 359718912 13505 33554432000 4194304 5683644 140733331201824 18446744073709551615 237278899782 0 0 16781312 514 16037507393361248514 0 0 17 0 0 0 0
/proc/1334/statm: 87822 13505 370 364 0 23728 0
[pid=1336] ppid=1334 vsize=268428 CPUtime=5.01 cores=0-7
/proc/1336/stat : 1336 (hybridGM3) R 1334 1336 1336 0 -1 4202496 8085 0 0 0 501 0 0 0 25 0 1 0 548251288 274870272 7820 33554432000 134512640 135204385 4288566960 18446744073709551615 134537401 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/1336/statm: 67107 7820 64 169 0 66937 0
[pid=1337] ppid=1316 vsize=361544 CPUtime=0 cores=0-7
/proc/1337/stat : 1337 (python) S 1316 1316 1221 0 -1 4202560 955 0 0 0 0 0 0 0 15 0 1 0 548251292 370221056 13509 33554432000 4194304 5683644 140733331201824 18446744073709551615 237278899782 0 0 16781312 514 4476007112488485121 0 0 17 4 0 0 0
/proc/1337/statm: 90386 13509 370 364 0 26289 0
[pid=1339] ppid=1337 vsize=268420 CPUtime=4.96 cores=0-7
/proc/1339/stat : 1339 (hybridGM3) R 1337 1339 1339 0 -1 4202496 8087 0 0 0 495 1 0 0 25 0 1 0 548251292 274862080 7822 33554432000 134512640 135204385 4286906864 18446744073709551615 134547687 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/1339/statm: 67105 7822 64 169 0 66935 0
[pid=1340] ppid=1316 vsize=371800 CPUtime=0 cores=0-7
/proc/1340/stat : 1340 (python) S 1316 1316 1221 0 -1 4202560 966 0 0 0 0 0 0 0 18 0 1 0 548251296 380723200 13513 33554432000 4194304 5683644 140733331201824 18446744073709551615 237278899782 0 0 16781312 514 16037503545070551297 0 0 17 4 0 0 0
/proc/1340/statm: 92950 13513 370 364 0 28850 0
[pid=1342] ppid=1340 vsize=465044 CPUtime=4.88 cores=0-7
/proc/1342/stat : 1342 (TNM) R 1340 1342 1342 0 -1 4202496 6014 0 0 0 487 1 0 0 25 0 1 0 548251296 476205056 5750 33554432000 134512640 135008748 4290187680 18446744073709551615 134529448 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/1342/statm: 116261 5750 46 122 0 115161 0
[pid=1343] ppid=1316 vsize=382056 CPUtime=0 cores=0-7
/proc/1343/stat : 1343 (python) S 1316 1316 1221 0 -1 4202560 1006 0 0 0 0 0 0 0 15 0 1 0 548251300 391225344 13523 33554432000 4194304 5683644 140733331201824 18446744073709551615 237278899782 0 0 16781312 514 16037499696779854082 0 0 17 0 0 0 0
/proc/1343/statm: 95514 13523 376 364 0 31411 0
[pid=1345] ppid=1343 vsize=268428 CPUtime=4.87 cores=0-7
/proc/1345/stat : 1345 (hybridGM3) R 1343 1345 1345 0 -1 4202496 8091 0 0 0 487 0 0 0 25 0 1 0 548251300 274870272 7822 33554432000 134512640 135204385 4293585008 18446744073709551615 134547802 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/1345/statm: 67107 7822 64 169 0 66937 0
[pid=1316/tid=1323] ppid=1314 vsize=392300 CPUtime=0 cores=0-7
/proc/1316/task/1323/stat : 1323 (python) S 1314 1316 1221 0 -1 4202560 27 934 0 0 0 0 0 0 25 0 9 0 548251272 401715200 15040 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290703281 0 0 16781312 2 18446604453897901888 0 0 -1 2 0 0 0
[pid=1316/tid=1326] ppid=1314 vsize=392300 CPUtime=0 cores=0-7
/proc/1316/task/1326/stat : 1326 (python) S 1314 1316 1221 0 -1 4202560 23 934 0 0 0 0 0 0 25 0 9 0 548251276 401715200 15040 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290703281 0 0 16781312 2 18446604453897901888 0 0 -1 0 0 0 0
[pid=1316/tid=1329] ppid=1314 vsize=392300 CPUtime=0 cores=0-7
/proc/1316/task/1329/stat : 1329 (python) S 1314 1316 1221 0 -1 4202560 22 934 0 0 0 0 0 0 20 0 9 0 548251280 401715200 15040 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290703281 0 0 16781312 2 18446604453897901888 0 0 -1 4 0 0 0
[pid=1316/tid=1332] ppid=1314 vsize=392300 CPUtime=0 cores=0-7
/proc/1316/task/1332/stat : 1332 (python) S 1314 1316 1221 0 -1 4202560 27 934 0 0 0 0 0 0 22 0 9 0 548251284 401715200 15040 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290703281 0 0 16781312 2 18446604453897901888 0 0 -1 0 0 0 0
[pid=1316/tid=1335] ppid=1314 vsize=392300 CPUtime=0 cores=0-7
/proc/1316/task/1335/stat : 1335 (python) S 1314 1316 1221 0 -1 4202560 26 934 0 0 0 0 0 0 24 0 9 0 548251288 401715200 15040 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290703281 0 0 16781312 2 18446604453897901888 0 0 -1 2 0 0 0
[pid=1316/tid=1338] ppid=1314 vsize=392300 CPUtime=0 cores=0-7
/proc/1316/task/1338/stat : 1338 (python) S 1314 1316 1221 0 -1 4202560 25 934 0 0 0 0 0 0 25 0 9 0 548251292 401715200 15040 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290703281 0 0 16781312 2 18446604453897901888 0 0 -1 0 0 0 0
[pid=1316/tid=1341] ppid=1314 vsize=392300 CPUtime=0 cores=0-7
/proc/1316/task/1341/stat : 1341 (python) S 1314 1316 1221 0 -1 4202560 24 934 0 0 0 0 0 0 25 0 9 0 548251296 401715200 15040 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290703281 0 0 16781312 2 18446604453897901888 0 0 -1 4 0 0 0
[pid=1316/tid=1344] ppid=1314 vsize=392300 CPUtime=0 cores=0-7
/proc/1316/task/1344/stat : 1344 (python) S 1314 1316 1221 0 -1 4202560 29 934 0 0 0 0 0 0 25 0 9 0 548251300 401715200 15040 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290703281 0 0 16781312 2 18446604453897901888 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 47.63
Current children cumulated vsize (KiB) 5505604

[startup+25.5014 s]
/proc/loadavg: 3.17 0.86 0.29 9/216 1346
/proc/meminfo: memFree=17736788/32950904 swapFree=67082284/67111528
[pid=1316] ppid=1314 vsize=392300 CPUtime=7.52 cores=0-7
/proc/1316/stat : 1316 (python) S 1314 1316 1221 0 -1 4202496 28808 934 1 0 736 16 0 0 25 0 9 0 548250520 401715200 15040 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290706379 0 0 16781312 2 18446744073709551615 0 0 17 0 0 0 0
/proc/1316/statm: 98075 15040 1889 364 0 33972 0
[pid=1322] ppid=1316 vsize=310264 CPUtime=0.03 cores=0-7
/proc/1322/stat : 1322 (python) S 1316 1316 1221 0 -1 4202560 915 0 0 0 1 2 0 0 15 0 1 0 548251272 317710336 13495 33554432000 4194304 5683644 140733331201824 18446744073709551615 237278899727 0 0 16781312 514 13867146202627145985 0 0 17 1 0 0 0
/proc/1322/statm: 77566 13495 376 364 0 13484 0
[pid=1324] ppid=1322 vsize=268428 CPUtime=17.96 cores=0-7
/proc/1324/stat : 1324 (hybridGM3) R 1322 1324 1324 0 -1 4202496 8084 0 0 0 1796 0 0 0 25 0 1 0 548251272 274870272 7821 33554432000 134512640 135204385 4286659248 18446744073709551615 134537403 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/1324/statm: 67107 7821 64 169 0 66937 0
[pid=1325] ppid=1316 vsize=320520 CPUtime=0.02 cores=0-7
/proc/1325/stat : 1325 (python) S 1316 1316 1221 0 -1 4202560 915 0 0 0 1 1 0 0 15 0 1 0 548251276 328212480 13493 33554432000 4194304 5683644 140733331201824 18446744073709551615 237278899782 0 0 16781312 514 13867146202627145988 0 0 17 4 0 0 0
/proc/1325/statm: 80130 13493 370 364 0 16045 0
[pid=1327] ppid=1325 vsize=268424 CPUtime=17.93 cores=0-7
/proc/1327/stat : 1327 (hybridGM3) R 1325 1327 1327 0 -1 4202496 8086 0 0 0 1792 1 0 0 25 0 1 0 548251276 274866176 7825 33554432000 134512640 135204385 4289545936 18446744073709551615 134537383 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/1327/statm: 67106 7825 64 169 0 66936 0
[pid=1328] ppid=1316 vsize=330776 CPUtime=0.02 cores=0-7
/proc/1328/stat : 1328 (python) S 1316 1316 1221 0 -1 4202560 913 0 0 0 1 1 0 0 15 0 1 0 548251280 338714624 13497 33554432000 4194304 5683644 140733331201824 18446744073709551615 237278899782 0 0 16781312 514 13867146202627145988 0 0 17 6 0 0 0
/proc/1328/statm: 82694 13497 370 364 0 18606 0
[pid=1330] ppid=1328 vsize=268428 CPUtime=17.87 cores=0-7
/proc/1330/stat : 1330 (hybridGM3) R 1328 1330 1330 0 -1 4202496 8087 0 0 0 1786 1 0 0 25 0 1 0 548251280 274870272 7823 33554432000 134512640 135204385 4293657904 18446744073709551615 134537573 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/1330/statm: 67107 7823 64 169 0 66937 0
[pid=1331] ppid=1316 vsize=341032 CPUtime=0.02 cores=0-7
/proc/1331/stat : 1331 (python) S 1316 1316 1221 0 -1 4202560 943 0 0 0 1 1 0 0 15 0 1 0 548251283 349216768 13501 33554432000 4194304 5683644 140733331201824 18446744073709551615 237278899782 0 0 16781312 514 13867146202627145985 0 0 17 4 0 0 0
/proc/1331/statm: 85258 13501 370 364 0 21167 0
[pid=1333] ppid=1331 vsize=268424 CPUtime=17.72 cores=0-7
/proc/1333/stat : 1333 (hybridGM3) R 1331 1333 1333 0 -1 4202496 8084 0 0 0 1771 1 0 0 25 0 1 0 548251284 274866176 7822 33554432000 134512640 135204385 4294121376 18446744073709551615 134537440 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/1333/statm: 67106 7822 64 169 0 66936 0
[pid=1334] ppid=1316 vsize=351288 CPUtime=0.01 cores=0-7
/proc/1334/stat : 1334 (python) S 1316 1316 1221 0 -1 4202560 950 0 0 0 1 0 0 0 15 0 1 0 548251287 359718912 13505 33554432000 4194304 5683644 140733331201824 18446744073709551615 237278899782 0 0 16781312 514 13867146202627145986 0 0 17 0 0 0 0
/proc/1334/statm: 87822 13505 370 364 0 23728 0
[pid=1336] ppid=1334 vsize=268428 CPUtime=17.81 cores=0-7
/proc/1336/stat : 1336 (hybridGM3) R 1334 1336 1336 0 -1 4202496 8089 0 0 0 1781 0 0 0 25 0 1 0 548251288 274870272 7824 33554432000 134512640 135204385 4288566960 18446744073709551615 134537573 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/1336/statm: 67107 7824 64 169 0 66937 0

################
# More data... #
################

[pid=1438] ppid=1316 vsize=621944 CPUtime=1.85 cores=0-7
/proc/1438/stat : 1438 (python) S 1316 1316 1221 0 -1 4202560 1299 0 0 0 100 85 0 0 18 0 2 0 548329772 636870656 13889 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290706379 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/1438/statm: 155486 13889 400 364 0 91332 0
[pid=1440] ppid=1438 vsize=431704 CPUtime=970.58 cores=0-7
/proc/1440/stat : 1440 (adaptg2wsat2009) T 1438 1440 1440 0 -1 4202496 6269 0 0 0 97055 3 0 0 25 0 1 0 548329773 442064896 5983 33554432000 134512640 135107411 4287680240 18446744073709551615 134531491 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/1440/statm: 107926 5983 59 146 0 106802 0
[pid=1438/tid=1464] ppid=1316 vsize=621944 CPUtime=0 cores=0-7
/proc/1438/task/1464/stat : 1464 (python) S 1316 1316 1221 0 -1 4202560 49 0 0 0 0 0 0 0 18 0 2 0 548349223 636870656 13889 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290703281 0 0 16781312 514 18446604453912540096 0 0 -1 1 0 0 0
[pid=1442] ppid=1316 vsize=632200 CPUtime=0.17 cores=0-7
/proc/1442/stat : 1442 (python) S 1316 1316 1221 0 -1 4202560 1278 0 0 0 9 8 0 0 19 0 2 0 548329882 647372800 13994 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290706379 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/1442/statm: 158050 13994 400 364 0 93893 0
[pid=1444] ppid=1442 vsize=465044 CPUtime=99.96 cores=0-7
/proc/1444/stat : 1444 (TNM) T 1442 1444 1444 0 -1 4202496 6030 0 0 0 9995 1 0 0 25 0 1 0 548329883 476205056 5749 33554432000 134512640 135008748 4292230128 18446744073709551615 134527729 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/1444/statm: 116261 5749 46 122 0 115161 0
[pid=1442/tid=1457] ppid=1316 vsize=632200 CPUtime=0 cores=0-7
/proc/1442/task/1457/stat : 1457 (python) S 1316 1316 1221 0 -1 4202560 196 0 0 0 0 0 0 0 19 0 2 0 548339899 647372800 13994 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290703281 0 0 16781312 514 18446604453897749568 0 0 -1 2 0 0 0
[pid=1449] ppid=1316 vsize=642456 CPUtime=0.91 cores=0-7
/proc/1449/stat : 1449 (python) S 1316 1316 1221 0 -1 4202560 1204 0 0 0 47 44 0 0 15 0 2 0 548330849 657874944 14073 33554432000 4194304 5683644 140733331201824 18446744073709551615 237278899782 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/1449/statm: 160614 14073 400 364 0 96454 0
[pid=1451] ppid=1449 vsize=268424 CPUtime=509.66 cores=0-7
/proc/1451/stat : 1451 (hybridGM3) R 1449 1451 1451 0 -1 4202496 8112 0 0 0 50965 1 0 0 25 0 1 0 548330850 274866176 7829 33554432000 134512640 135204385 4291573264 18446744073709551615 134536843 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/1451/statm: 67106 7829 64 169 0 66936 0
[pid=1449/tid=1465] ppid=1316 vsize=642456 CPUtime=0 cores=0-7
/proc/1449/task/1465/stat : 1465 (python) S 1316 1316 1221 0 -1 4202560 104 0 0 0 0 0 0 0 15 0 2 0 548350495 657874944 14073 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290703281 0 0 16781312 514 18446604453937961088 0 0 -1 1 0 0 0
[pid=1452] ppid=1316 vsize=652712 CPUtime=0.36 cores=0-7
/proc/1452/stat : 1452 (python) S 1316 1316 1221 0 -1 4202560 1211 0 0 0 19 17 0 0 19 0 2 0 548330874 668377088 14010 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290706379 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/1452/statm: 163178 14010 400 364 0 99015 0
[pid=1454] ppid=1452 vsize=431704 CPUtime=195.88 cores=0-7
/proc/1454/stat : 1454 (adaptg2wsat2009) T 1452 1454 1454 0 -1 4202496 6269 0 0 0 19587 1 0 0 25 0 1 0 548330875 442064896 5983 33554432000 134512640 135107411 4288034848 18446744073709551615 134532074 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/1454/statm: 107926 5983 59 146 0 106802 0
[pid=1452/tid=1467] ppid=1316 vsize=652712 CPUtime=0 cores=0-7
/proc/1452/task/1467/stat : 1467 (python) S 1316 1316 1221 0 -1 4202560 101 0 0 0 0 0 0 0 19 0 2 0 548350910 668377088 14010 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290703281 0 0 16781312 514 18446604453925345280 0 0 -1 3 0 0 0
[pid=1458] ppid=1316 vsize=662968 CPUtime=0.18 cores=0-7
/proc/1458/stat : 1458 (python) S 1316 1316 1221 0 -1 4202560 1256 0 0 0 10 8 0 0 19 0 2 0 548340508 678879232 14080 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290706379 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/1458/statm: 165742 14080 406 364 0 101576 0
[pid=1460] ppid=1458 vsize=260404 CPUtime=98.71 cores=0-7
/proc/1460/stat : 1460 (FH) T 1458 1460 1460 0 -1 4202496 6192 0 0 0 9871 0 0 0 25 0 1 0 548340509 266653696 5899 33554432000 134512640 134936988 4293338624 18446744073709551615 134547038 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/1460/statm: 65101 5899 41 104 0 64605 0
[pid=1458/tid=1466] ppid=1316 vsize=662968 CPUtime=0 cores=0-7
/proc/1458/task/1466/stat : 1466 (python) S 1316 1316 1221 0 -1 4202560 107 0 0 0 0 0 0 0 19 0 2 0 548350527 678879232 14080 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290703281 0 0 16781312 514 18446604453937961984 0 0 -1 7 0 0 0
[pid=1470] ppid=1316 vsize=673224 CPUtime=2.04 cores=0-7
/proc/1470/stat : 1470 (python) S 1316 1316 1221 0 -1 4202560 1262 0 0 0 107 97 0 0 18 0 2 0 548360031 689381376 14032 33554432000 4194304 5683644 140733331201824 18446744073709551615 237278899782 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/1470/statm: 168306 14032 407 364 0 104137 0
[pid=1472] ppid=1470 vsize=465044 CPUtime=1107.36 cores=0-7
/proc/1472/stat : 1472 (TNM) R 1470 1472 1472 0 -1 4202496 6037 0 0 0 110734 2 0 0 25 0 1 0 548360032 476205056 5749 33554432000 134512640 135008748 4286877264 18446744073709551615 134527773 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/1472/statm: 116261 5749 46 122 0 115161 0
[pid=1470/tid=1476] ppid=1316 vsize=673224 CPUtime=0 cores=0-7
/proc/1470/task/1476/stat : 1476 (python) S 1316 1316 1221 0 -1 4202560 111 0 0 0 0 0 0 0 19 0 2 0 548370049 689381376 14032 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290703281 0 0 16781312 514 18446604453984703488 0 0 -1 1 0 0 0
[pid=1488] ppid=1316 vsize=683480 CPUtime=0.55 cores=0-7
/proc/1488/stat : 1488 (python) S 1316 1316 1221 0 -1 4202560 1277 0 0 0 27 28 0 0 15 0 2 0 548409148 699883520 14067 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290706379 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/1488/statm: 170870 14067 406 364 0 106698 0
[pid=1490] ppid=1488 vsize=268420 CPUtime=296.66 cores=0-7
/proc/1490/stat : 1490 (hybridGM3) T 1488 1490 1490 0 -1 4202496 8113 0 0 0 29665 1 0 0 25 0 1 0 548409149 274862080 7824 33554432000 134512640 135204385 4286853776 18446744073709551615 134537113 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/1490/statm: 67105 7824 64 169 0 66935 0
[pid=1488/tid=1502] ppid=1316 vsize=683480 CPUtime=0 cores=0-7
/proc/1488/task/1502/stat : 1502 (python) S 1316 1316 1221 0 -1 4202560 109 0 0 0 0 0 0 0 15 0 2 0 548428796 699883520 14067 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290703281 0 0 16781312 514 18446604442907938880 0 0 -1 1 0 0 0
[pid=1491] ppid=1316 vsize=693736 CPUtime=1.46 cores=0-7
/proc/1491/stat : 1491 (python) S 1316 1316 1221 0 -1 4202560 1271 0 0 0 76 70 0 0 18 0 2 0 548409160 710385664 14165 33554432000 4194304 5683644 140733331201824 18446744073709551615 237278899782 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/1491/statm: 173434 14165 406 364 0 109259 0
[pid=1493] ppid=1491 vsize=433188 CPUtime=776.59 cores=0-7
/proc/1493/stat : 1493 (NCVWr) R 1491 1493 1493 0 -1 4202496 6436 0 0 0 77658 1 0 0 25 0 1 0 548409160 443584512 6148 33554432000 134512640 135113558 4288204288 18446744073709551615 134531867 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/1493/statm: 108297 6148 58 147 0 107172 0
[pid=1491/tid=1512] ppid=1316 vsize=693736 CPUtime=0 cores=0-7
/proc/1491/task/1512/stat : 1512 (python) S 1316 1316 1221 0 -1 4202560 103 0 0 0 0 0 0 0 19 0 2 0 548459156 710385664 14165 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290703281 0 0 16781312 514 18446604452255689472 0 0 -1 7 0 0 0
[pid=1498] ppid=1316 vsize=703992 CPUtime=0.18 cores=0-7
/proc/1498/stat : 1498 (python) S 1316 1316 1221 0 -1 4202560 1262 0 0 0 10 8 0 0 19 0 2 0 548427702 720887808 14048 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290706379 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/1498/statm: 175998 14048 400 364 0 111820 0
[pid=1500] ppid=1498 vsize=260404 CPUtime=99 cores=0-7
/proc/1500/stat : 1500 (FH) T 1498 1500 1500 0 -1 4202496 6191 0 0 0 9899 1 0 0 25 0 1 0 548427702 266653696 5897 33554432000 134512640 134936988 4292266752 18446744073709551615 134553566 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/1500/statm: 65101 5897 41 104 0 64605 0
[pid=1498/tid=1505] ppid=1316 vsize=703992 CPUtime=0 cores=0-7
/proc/1498/task/1505/stat : 1505 (python) S 1316 1316 1221 0 -1 4202560 108 0 0 0 0 0 0 0 19 0 2 0 548437623 720887808 14048 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290703281 0 0 16781312 514 18446604452255688576 0 0 -1 0 0 0 0
[pid=1520] ppid=1316 vsize=714248 CPUtime=0.04 cores=0-7
/proc/1520/stat : 1520 (python) S 1316 1316 1221 0 -1 4202560 1154 0 0 0 2 2 0 0 15 0 1 0 548486340 731389952 13896 33554432000 4194304 5683644 140733331201824 18446744073709551615 237278899782 0 0 16781312 514 13867146202627145986 0 0 17 4 0 0 0
/proc/1520/statm: 178562 13896 372 364 0 114381 0
[pid=1522] ppid=1520 vsize=268424 CPUtime=26.36 cores=0-7
/proc/1522/stat : 1522 (hybridGM3) R 1520 1522 1522 0 -1 4202496 8112 0 0 0 2635 1 0 0 25 0 1 0 548486341 274866176 7819 33554432000 134512640 135204385 4291332944 18446744073709551615 134536986 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/1522/statm: 67106 7819 64 169 0 66936 0
[pid=1524] ppid=1316 vsize=724504 CPUtime=0 cores=0-7
/proc/1524/stat : 1524 (python) S 1316 1316 1221 0 -1 4202560 964 0 0 0 0 0 0 0 19 0 1 0 548488976 741892096 13873 33554432000 4194304 5683644 140733331201824 18446744073709551615 237278899782 0 0 16781312 514 17164767121209556991 0 0 17 4 0 0 0
/proc/1524/statm: 181126 13873 345 364 0 116942 0
[pid=1526] ppid=1524 vsize=422384 CPUtime=0.14 cores=0-7
/proc/1526/stat : 1526 (NCVWr) R 1524 1526 1526 0 -1 4202496 3734 0 0 0 14 0 0 0 21 0 1 0 548488977 432521216 3440 33554432000 134512640 135113558 4287499360 18446744073709551615 134526604 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/1526/statm: 105596 3440 55 147 0 104471 0
[pid=1316/tid=1323] ppid=1314 vsize=734748 CPUtime=0 cores=0-7
/proc/1316/task/1323/stat : 1323 (python) S 1314 1316 1221 0 -1 4202560 45 934 0 0 0 0 0 0 15 0 36 0 548251272 752381952 15476 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290703281 0 0 16781312 2 18446604453897901888 0 0 -1 2 0 0 0
[pid=1316/tid=1326] ppid=1314 vsize=734748 CPUtime=0 cores=0-7
/proc/1316/task/1326/stat : 1326 (python) S 1314 1316 1221 0 -1 4202560 42 934 0 0 0 0 0 0 15 0 36 0 548251276 752381952 15476 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290703281 0 0 16781312 2 18446604453897901888 0 0 -1 6 0 0 0
[pid=1316/tid=1329] ppid=1314 vsize=734748 CPUtime=0 cores=0-7
/proc/1316/task/1329/stat : 1329 (python) S 1314 1316 1221 0 -1 4202560 34 934 0 0 0 0 0 0 18 0 36 0 548251280 752381952 15476 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290703281 0 0 16781312 2 18446604453897901888 0 0 -1 4 0 0 0
[pid=1316/tid=1332] ppid=1314 vsize=734748 CPUtime=0 cores=0-7
/proc/1316/task/1332/stat : 1332 (python) S 1314 1316 1221 0 -1 4202560 40 934 0 0 0 0 0 0 18 0 36 0 548251284 752381952 15476 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290703281 0 0 16781312 2 18446604453897901888 0 0 -1 0 0 0 0
[pid=1316/tid=1335] ppid=1314 vsize=734748 CPUtime=0 cores=0-7
/proc/1316/task/1335/stat : 1335 (python) S 1314 1316 1221 0 -1 4202560 43 934 0 0 0 0 0 0 15 0 36 0 548251288 752381952 15476 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290703281 0 0 16781312 2 18446604453897901888 0 0 -1 2 0 0 0
[pid=1316/tid=1338] ppid=1314 vsize=734748 CPUtime=0 cores=0-7
/proc/1316/task/1338/stat : 1338 (python) S 1314 1316 1221 0 -1 4202560 40 934 0 0 0 0 0 0 18 0 36 0 548251292 752381952 15476 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290703281 0 0 16781312 2 18446604453897901888 0 0 -1 0 0 0 0
[pid=1316/tid=1341] ppid=1314 vsize=734748 CPUtime=0 cores=0-7
/proc/1316/task/1341/stat : 1341 (python) S 1314 1316 1221 0 -1 4202560 27 934 0 0 0 0 0 0 18 0 36 0 548251296 752381952 15476 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290703281 0 0 16781312 2 18446604453897901888 0 0 -1 4 0 0 0
[pid=1316/tid=1344] ppid=1314 vsize=734748 CPUtime=0 cores=0-7
/proc/1316/task/1344/stat : 1344 (python) S 1314 1316 1221 0 -1 4202560 41 934 0 0 0 0 0 0 18 0 36 0 548251300 752381952 15476 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290703281 0 0 16781312 2 18446604453897901888 0 0 -1 0 0 0 0
[pid=1316/tid=1354] ppid=1314 vsize=734748 CPUtime=0 cores=0-7
/proc/1316/task/1354/stat : 1354 (python) S 1314 1316 1221 0 -1 4202560 44 934 0 0 0 0 0 0 18 0 36 0 548261316 752381952 15476 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290703281 0 0 16781312 2 18446604453897901888 0 0 -1 0 0 0 0
[pid=1316/tid=1369] ppid=1314 vsize=734748 CPUtime=0 cores=0-7
/proc/1316/task/1369/stat : 1369 (python) S 1314 1316 1221 0 -1 4202560 55 934 0 0 0 0 0 0 15 0 36 0 548280446 752381952 15476 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290703281 0 0 16781312 2 18446604453897901888 0 0 -1 1 0 0 0
[pid=1316/tid=1372] ppid=1314 vsize=734748 CPUtime=0 cores=0-7
/proc/1316/task/1372/stat : 1372 (python) S 1314 1316 1221 0 -1 4202560 104 934 0 0 0 0 0 0 18 0 36 0 548280935 752381952 15476 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290703281 0 0 16781312 2 18446604453897901888 0 0 -1 0 0 0 0
[pid=1316/tid=1375] ppid=1314 vsize=734748 CPUtime=0 cores=0-7
/proc/1316/task/1375/stat : 1375 (python) S 1314 1316 1221 0 -1 4202560 38 934 0 0 0 0 0 0 18 0 36 0 548280941 752381952 15476 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290703281 0 0 16781312 2 18446604453897901888 0 0 -1 1 0 0 0
[pid=1316/tid=1382] ppid=1314 vsize=734748 CPUtime=0 cores=0-7
/proc/1316/task/1382/stat : 1382 (python) S 1314 1316 1221 0 -1 4202560 52 934 0 0 0 0 0 0 18 0 36 0 548290465 752381952 15476 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290703281 0 0 16781312 2 18446604453897901888 0 0 -1 1 0 0 0
[pid=1316/tid=1385] ppid=1314 vsize=734748 CPUtime=0 cores=0-7
/proc/1316/task/1385/stat : 1385 (python) S 1314 1316 1221 0 -1 4202560 108 934 0 0 0 0 0 0 18 0 36 0 548290561 752381952 15476 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290703281 0 0 16781312 2 18446604453897901888 0 0 -1 0 0 0 0
[pid=1316/tid=1389] ppid=1314 vsize=734748 CPUtime=0 cores=0-7
/proc/1316/task/1389/stat : 1389 (python) S 1314 1316 1221 0 -1 4202560 48 934 0 0 0 0 0 0 18 0 36 0 548290959 752381952 15476 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290703281 0 0 16781312 2 18446604453897901888 0 0 -1 0 0 0 0
[pid=1316/tid=1395] ppid=1314 vsize=734748 CPUtime=0 cores=0-7
/proc/1316/task/1395/stat : 1395 (python) S 1314 1316 1221 0 -1 4202560 102 934 0 0 0 0 0 0 19 0 36 0 548300102 752381952 15476 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290703281 0 0 16781312 2 18446604453897901888 0 0 -1 2 0 0 0
[pid=1316/tid=1400] ppid=1314 vsize=734748 CPUtime=0 cores=0-7
/proc/1316/task/1400/stat : 1400 (python) S 1314 1316 1221 0 -1 4202560 32 934 0 0 0 0 0 0 18 0 36 0 548300582 752381952 15476 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290703281 0 0 16781312 2 18446604453897901888 0 0 -1 2 0 0 0
[pid=1316/tid=1407] ppid=1314 vsize=734748 CPUtime=0 cores=0-7
/proc/1316/task/1407/stat : 1407 (python) S 1314 1316 1221 0 -1 4202560 36 934 0 0 0 0 0 0 20 0 36 0 548310024 752381952 15476 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290703281 0 0 16781312 2 18446604453897901888 0 0 -1 1 0 0 0
[pid=1316/tid=1411] ppid=1314 vsize=734748 CPUtime=0 cores=0-7
/proc/1316/task/1411/stat : 1411 (python) S 1314 1316 1221 0 -1 4202560 139 934 0 0 0 0 0 0 15 0 36 0 548310117 752381952 15476 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290703281 0 0 16781312 2 18446604453897901888 0 0 -1 2 0 0 0
[pid=1316/tid=1415] ppid=1314 vsize=734748 CPUtime=0 cores=0-7
/proc/1316/task/1415/stat : 1415 (python) S 1314 1316 1221 0 -1 4202560 42 934 0 0 0 0 0 0 19 0 36 0 548310720 752381952 15476 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290703281 0 0 16781312 2 18446604453897901888 0 0 -1 7 0 0 0
[pid=1316/tid=1418] ppid=1314 vsize=734748 CPUtime=0 cores=0-7
/proc/1316/task/1418/stat : 1418 (python) S 1314 1316 1221 0 -1 4202560 43 934 0 0 0 0 0 0 18 0 36 0 548310728 752381952 15476 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290703281 0 0 16781312 2 18446604453897901888 0 0 -1 2 0 0 0
[pid=1316/tid=1425] ppid=1314 vsize=734748 CPUtime=0 cores=0-7
/proc/1316/task/1425/stat : 1425 (python) S 1314 1316 1221 0 -1 4202560 37 934 0 0 0 0 0 0 20 0 36 0 548319949 752381952 15476 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290703281 0 0 16781312 2 18446604453897901888 0 0 -1 1 0 0 0
[pid=1316/tid=1428] ppid=1314 vsize=734748 CPUtime=0 cores=0-7
/proc/1316/task/1428/stat : 1428 (python) S 1314 1316 1221 0 -1 4202560 40 934 0 0 0 0 0 0 18 0 36 0 548320143 752381952 15476 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290703281 0 0 16781312 2 18446604453897901888 0 0 -1 2 0 0 0
[pid=1316/tid=1432] ppid=1314 vsize=734748 CPUtime=0 cores=0-7
/proc/1316/task/1432/stat : 1432 (python) S 1314 1316 1221 0 -1 4202560 37 934 0 0 0 0 0 0 20 0 36 0 548320220 752381952 15476 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290703281 0 0 16781312 2 18446604453897901888 0 0 -1 2 0 0 0
[pid=1316/tid=1439] ppid=1314 vsize=734748 CPUtime=0 cores=0-7
/proc/1316/task/1439/stat : 1439 (python) S 1314 1316 1221 0 -1 4202560 59 934 0 0 0 0 0 0 18 0 36 0 548329773 752381952 15476 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290703281 0 0 16781312 2 18446604453897901888 0 0 -1 1 0 0 0
[pid=1316/tid=1443] ppid=1314 vsize=734748 CPUtime=0 cores=0-7
/proc/1316/task/1443/stat : 1443 (python) S 1314 1316 1221 0 -1 4202560 151 934 0 0 0 0 0 0 20 0 36 0 548329883 752381952 15476 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290703281 0 0 16781312 2 18446604453897901888 0 0 -1 1 0 0 0
[pid=1316/tid=1450] ppid=1314 vsize=734748 CPUtime=0 cores=0-7
/proc/1316/task/1450/stat : 1450 (python) S 1314 1316 1221 0 -1 4202560 52 934 0 0 0 0 0 0 18 0 36 0 548330850 752381952 15476 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290703281 0 0 16781312 2 18446604453897901888 0 0 -1 1 0 0 0
[pid=1316/tid=1453] ppid=1314 vsize=734748 CPUtime=0 cores=0-7
/proc/1316/task/1453/stat : 1453 (python) S 1314 1316 1221 0 -1 4202560 35 934 0 0 0 0 0 0 20 0 36 0 548330875 752381952 15476 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290703281 0 0 16781312 2 18446604453897901888 0 0 -1 2 0 0 0
[pid=1316/tid=1459] ppid=1314 vsize=734748 CPUtime=0 cores=0-7
/proc/1316/task/1459/stat : 1459 (python) S 1314 1316 1221 0 -1 4202560 39 934 0 0 0 0 0 0 19 0 36 0 548340509 752381952 15476 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290703281 0 0 16781312 2 18446604453897901888 0 0 -1 2 0 0 0
[pid=1316/tid=1471] ppid=1314 vsize=734748 CPUtime=0 cores=0-7
/proc/1316/task/1471/stat : 1471 (python) S 1314 1316 1221 0 -1 4202560 48 934 0 0 0 0 0 0 18 0 36 0 548360032 752381952 15476 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290703281 0 0 16781312 2 18446604453897901888 0 0 -1 2 0 0 0
[pid=1316/tid=1489] ppid=1314 vsize=734748 CPUtime=0 cores=0-7
/proc/1316/task/1489/stat : 1489 (python) S 1314 1316 1221 0 -1 4202560 46 934 0 0 0 0 0 0 18 0 36 0 548409148 752381952 15476 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290703281 0 0 16781312 2 18446604453897901888 0 0 -1 2 0 0 0
[pid=1316/tid=1492] ppid=1314 vsize=734748 CPUtime=0 cores=0-7
/proc/1316/task/1492/stat : 1492 (python) S 1314 1316 1221 0 -1 4202560 37 934 0 0 0 0 0 0 18 0 36 0 548409160 752381952 15476 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290703281 0 0 16781312 2 18446604453897901888 0 0 -1 1 0 0 0
[pid=1316/tid=1499] ppid=1314 vsize=734748 CPUtime=0 cores=0-7
/proc/1316/task/1499/stat : 1499 (python) S 1314 1316 1221 0 -1 4202560 37 934 0 0 0 0 0 0 20 0 36 0 548427702 752381952 15476 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290703281 0 0 16781312 2 18446604453897901888 0 0 -1 1 0 0 0
[pid=1316/tid=1521] ppid=1314 vsize=734748 CPUtime=0 cores=0-7
/proc/1316/task/1521/stat : 1521 (python) S 1314 1316 1221 0 -1 4202560 31 934 0 0 0 0 0 0 20 0 36 0 548486340 752381952 15476 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290703281 0 0 16781312 2 18446604453897901888 0 0 -1 4 0 0 0
[pid=1316/tid=1525] ppid=1314 vsize=734748 CPUtime=0 cores=0-7
/proc/1316/task/1525/stat : 1525 (python) S 1314 1316 1221 0 -1 4202560 26 934 0 0 0 0 0 0 20 0 36 0 548488977 752381952 15476 33554432000 4194304 5683644 140733331201824 18446744073709551615 237290703281 0 0 16781312 2 18446604453897901888 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 18933.6
Current children cumulated vsize (KiB) 31826220

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 1316 and gives
#  childrusage.ru_utime.tv_sec=4459
#  childrusage.ru_utime.tv_usec=333078
#  childrusage.ru_stime.tv_sec=16
#  childrusage.ru_stime.tv_usec=547484
# CPU time returned by wait4() is 4475.88
# while last known CPU time is 18933.6
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 2384.92
CPU time (s): 18933.6
CPU user time (s): 18917
CPU system time (s): 16.68
CPU usage (%): 793.891
Max. virtual memory (cumulated for all children) (KiB): 31826220

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4459.33
system time used= 16.5475
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 198573
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 25246
involuntary context switches= 35755

runsolver used 24.1533 second user time and 63.6273 second system time

The end

Launcher Data 

Begin job on node121 at 2011-05-02 22:01:48
IDJOB=3321730
IDBENCH=82508
IDSOLVER=1712
FILE ID=node121/3321730-1304366507
RUNJOBID= node121-1304366467-1235
PBS_JOBID= 13181750
Free space on /tmp= 73512 MiB

SOLVER NAME= borg-sat 11.03.02-random
BENCH NAME= SAT11/random/large/unif-k3-r4.2-v35000-c147000-S883457431-033-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3321730-1304366507/watcher-3321730-1304366507 -o /tmp/evaluation-result-3321730-1304366507/solver-3321730-1304366507 -C 40000 -W 5100 -M 31000  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3321730-1304366507.cnf -seed 1397071266 -budget 40000 -cores 8

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 0d0abbf84f73b4226aa3bedf08367d2e
RANDOM SEED=1397071266

node121.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.817
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.63
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.817
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.817
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.84
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.817
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.817
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.817
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.817
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.817
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950904 kB
MemFree:      18067820 kB
Buffers:       2196700 kB
Cached:        9666748 kB
SwapCached:       2776 kB
Active:       11343884 kB
Inactive:      3015572 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950904 kB
LowFree:      18067820 kB
SwapTotal:    67111528 kB
SwapFree:     67082284 kB
Dirty:           34892 kB
Writeback:           0 kB
AnonPages:     2494304 kB
Mapped:          17904 kB
Slab:           451156 kB
PageTables:      10260 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586980 kB
Committed_AS:  2845544 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73480 MiB
End job on node121 at 2011-05-02 22:41:35