Trace number 3385418

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-par-random (fixed)? (MO) 9590.3 1353.12

General information on the benchmark

Namerandom/large/
unif-k5-r20-v1500-c30000-S1305346115-067-UNKNOWN.cnf
MD5SUM0c1f90cdbd54dc48a7f7e4333f537217
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark1107.92
Satisfiable
(Un)Satisfiability was proved
Number of variables1500
Number of clauses30000
Sum of the clauses size150000
Maximum clause length5
Minimum clause length5
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 530000
Number of clauses of size over 50

Solver Data 

0.36/0.67	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.36/0.67	c INFO: solving HOME/instance-3385418-1305566412.cnf
0.36/0.67	c INFO: computing task features
0.53/0.82	c INFO: computed variable-clause graph statistics
0.53/0.82	c INFO: computed clause balance statistics
1.33/1.61	c INFO: computed variable balance statistics
1.73/2.02	c INFO: computed small-clause counts
1.73/2.03	c INFO: computed Horn-clause counts
1.73/2.06	c INFO: computed variable graph statistics
1.73/2.06	c INFO: collected features for HOME/instance-3385418-1305566412.cnf in 1.39 s
1.73/2.09	c INFO: running gnovelty+2@0 for 97 with 4998 remaining (b = 0.38)
1.86/2.12	c INFO: running gnovelty+2@0 for 97 with 4998 remaining (b = 0.14)
1.86/2.15	c INFO: running adaptg2wsat2009++@0 for 97 with 4998 remaining (b = 0.09)
1.86/2.18	c INFO: running adaptg2wsat2009++@0 for 97 with 4998 remaining (b = 0.07)
2.05/2.21	c INFO: running adaptg2wsat2009++@0 for 97 with 4998 remaining (b = 0.06)
2.05/2.25	c INFO: running gnovelty+2@0 for 97 with 4998 remaining (b = 0.05)
2.05/2.28	c INFO: running FH@0 for 97 with 4998 remaining (b = 0.04)
2.23/2.31	c INFO: running adaptg2wsat2009++@0 for 97 with 4998 remaining (b = 0.03)
702.00/102.42	c INFO: running adaptg2wsat2009++@0 for 97 with 4898 remaining (b = 0.03)
702.67/102.51	c INFO: running FH@0 for 97 with 4898 remaining (b = 0.02)
703.25/102.65	c INFO: running hybridGM3@0 for 194 with 4898 remaining (b = 0.04)
745.28/108.62	c INFO: running gnovelty+2@0 for 97 with 4892 remaining (b = 0.02)
785.78/114.50	c INFO: running hybridGM3@0 for 194 with 4886 remaining (b = 0.03)
870.22/126.55	c INFO: running hybridGM3@0 for 194 with 4874 remaining (b = 0.03)
912.01/132.54	c INFO: running TNM@0 for 1552 with 4868 remaining (b = 0.21)
997.10/144.79	c INFO: running adaptg2wsat2009++@0 for 97 with 4855 remaining (b = 0.02)
1493.38/215.04	c INFO: running FH@0 for 97 with 4785 remaining (b = 0.01)
1533.10/220.80	c INFO: running hybridGM3@0 for 194 with 4779 remaining (b = 0.02)
1576.32/226.90	c INFO: running adaptg2wsat2009++@0 for 97 with 4773 remaining (b = 0.01)
1703.14/245.14	c INFO: running adaptg2wsat2009++@100 for 97 with 4755 remaining (b = 0.01)
2080.24/299.30	c INFO: running hybridGM3@102 for 97 with 4701 remaining (b = 0.01)
2254.39/324.27	c INFO: running gnovelty+2@0 for 97 with 4676 remaining (b = 0.01)
2290.58/329.47	c INFO: running hybridGM3@100 for 97 with 4671 remaining (b = 0.01)
2360.27/339.44	c INFO: running hybridGM3@0 for 194 with 4661 remaining (b = 0.02)
2422.97/348.48	c INFO: running hybridGM3@0 for 194 with 4652 remaining (b = 0.02)
2701.68/388.43	c INFO: running hybridGM3@98 for 97 with 4612 remaining (b = 0.01)
3000.06/430.72	c INFO: running TNM@0 for 1455 with 4569 remaining (b = 0.13)
3000.70/430.80	c INFO: running adaptg2wsat2009++@0 for 97 with 4569 remaining (b = 0.01)
3007.71/431.88	c INFO: running adaptg2wsat2009++@0 for 97 with 4568 remaining (b = 0.01)
3305.28/471.65	c INFO: running hybridGM3@102 for 97 with 4529 remaining (b = 0.01)
3530.07/501.79	c INFO: running adaptg2wsat2009++@99 for 97 with 4498 remaining (b = 0.01)
3719.95/528.13	c INFO: running iPAWS@0 for 97 with 4472 remaining (b = 0.01)
3817.91/542.11	c INFO: running hybridGM3@97 for 97 with 4458 remaining (b = 0.01)
4015.80/569.36	c INFO: running hybridGM3@0 for 194 with 4431 remaining (b = 0.02)
4125.05/585.03	c INFO: running FH@0 for 291 with 4415 remaining (b = 0.03)
4211.35/597.42	c INFO: running adaptg2wsat2009++@97 for 97 with 4403 remaining (b = 0.01)
4223.25/599.14	c INFO: running FH@99 for 291 with 4401 remaining (b = 0.03)
4554.11/643.46	c INFO: running adaptg2wsat2009++@296 for 291 with 4357 remaining (b = 0.03)
4621.93/652.54	c INFO: running hybridGM3@0 for 194 with 4348 remaining (b = 0.02)
4930.05/694.76	c INFO: running TNM@0 for 1552 with 4305 remaining (b = 0.12)
5553.16/784.17	c INFO: running hybridGM3@97 for 97 with 4216 remaining (b = 0.01)
6288.98/885.52	c INFO: running iPAWS@0 for 97 with 4115 remaining (b = 0.01)
6298.63/886.94	c INFO: running FH@195 for 194 with 4113 remaining (b = 0.02)
6480.38/913.10	c INFO: running gnovelty+2@0 for 97 with 4087 remaining (b = 0.01)
6539.48/921.53	c INFO: running hybridGM3@96 for 97 with 4079 remaining (b = 0.01)
7028.82/991.89	c INFO: running hybridGM3@0 for 194 with 4008 remaining (b = 0.02)
7040.63/993.52	c INFO: running adaptg2wsat2009++@397 for 194 with 4007 remaining (b = 0.01)
7179.14/1013.49	c INFO: running hybridGM3@0 for 194 with 3987 remaining (b = 0.01)
7202.80/1016.87	c INFO: running hybridGM3@0 for 194 with 3983 remaining (b = 0.01)
7912.82/1114.65	c INFO: running NCVWr@0 for 485 with 3886 remaining (b = 0.03)
8480.81/1194.61	c INFO: running hybridGM3@291 for 194 with 3806 remaining (b = 0.02)
8640.32/1216.21	c INFO: running hybridGM3@94 for 97 with 3784 remaining (b = 0.01)
8747.30/1231.61	c INFO: running FH@194 for 194 with 3769 remaining (b = 0.01)
8758.41/1233.26	c INFO: running adaptg2wsat2009++@497 for 97 with 3767 remaining (b = 0.01)
9546.32/1346.67	c INFO: running FH@0 for 97 with 3654 remaining (b = 0.01)
9588.04/1352.66	c INFO: running iPAWS@0 for 97 with 3648 remaining (b = 0.01)
9590.30/1353.10	c INFO: running hybridGM3@0 for 194 with 3647 remaining (b = 0.01)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3385418-1305566412/watcher-3385418-1305566412 -o /tmp/evaluation-result-3385418-1305566412/solver-3385418-1305566412 -C 40000 -W 5100 -M 31000 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3385418-1305566412.cnf -seed 1374732334 -budget 40000 -cores 8 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 9.29 8.82 6.78 2/348 25711
/proc/meminfo: memFree=8155492/32950928 swapFree=64721852/67111528
[pid=25711] ppid=25709 vsize=15776 CPUtime=0 cores=0-7
/proc/25711/stat : 25711 (python) R 25709 25711 25311 0 -1 4202496 900 0 0 0 0 0 0 0 25 0 1 0 668241029 16154624 782 33554432000 4194304 5683644 140737118563152 18446744073709551615 4426576 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/25711/statm: 3944 782 345 364 0 431 0

[startup+0.060899 s]
/proc/loadavg: 9.29 8.82 6.78 2/348 25711
/proc/meminfo: memFree=8155492/32950928 swapFree=64721852/67111528
[pid=25711] ppid=25709 vsize=29792 CPUtime=0.03 cores=0-7
/proc/25711/stat : 25711 (python) R 25709 25711 25311 0 -1 4202496 1454 0 0 0 3 0 0 0 25 0 1 0 668241029 30507008 1296 33554432000 4194304 5683644 140737118563152 18446744073709551615 248089696256 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/25711/statm: 7448 1296 443 364 0 818 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 29792

[startup+0.100891 s]
/proc/loadavg: 9.29 8.82 6.78 2/348 25711
/proc/meminfo: memFree=8155492/32950928 swapFree=64721852/67111528
[pid=25711] ppid=25709 vsize=89772 CPUtime=0.06 cores=0-7
/proc/25711/stat : 25711 (python) S 25709 25711 25311 0 -1 4202496 2671 0 0 0 5 1 0 0 25 0 1 0 668241029 91926528 2388 33554432000 4194304 5683644 140737118563152 18446744073709551615 248089696256 0 2147483391 16781312 2 18446744071699199166 0 0 17 4 0 0 0
/proc/25711/statm: 22443 2388 750 364 0 1623 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 89772

[startup+0.300857 s]
/proc/loadavg: 9.29 8.82 6.78 2/348 25711
/proc/meminfo: memFree=8155492/32950928 swapFree=64721852/67111528
[pid=25711] ppid=25709 vsize=135752 CPUtime=0.17 cores=0-7
/proc/25711/stat : 25711 (python) R 25709 25711 25311 0 -1 4202496 5962 934 1 0 13 4 0 0 25 0 1 0 668241029 139010048 4151 33554432000 4194304 5683644 140737118563152 18446744073709551615 248089694213 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/25711/statm: 33938 4151 1032 364 0 3124 0
Current children cumulated CPU time (s) 0.17
Current children cumulated vsize (KiB) 135752

[startup+0.700786 s]
/proc/loadavg: 9.29 8.82 6.78 2/348 25711
/proc/meminfo: memFree=8155492/32950928 swapFree=64721852/67111528
[pid=25711] ppid=25709 vsize=280508 CPUtime=0.43 cores=0-7
/proc/25711/stat : 25711 (python) R 25709 25711 25311 0 -1 4202496 9790 934 1 0 32 11 0 0 25 0 1 0 668241029 287240192 7474 33554432000 4194304 5683644 140737118563152 18446744073709551615 4445115 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/25711/statm: 70127 7474 1761 364 0 6051 0
Current children cumulated CPU time (s) 0.43
Current children cumulated vsize (KiB) 280508

[startup+1.50064 s]
/proc/loadavg: 9.29 8.82 6.78 3/349 25716
/proc/meminfo: memFree=8123740/32950928 swapFree=64721852/67111528
[pid=25711] ppid=25709 vsize=287584 CPUtime=1.23 cores=0-7
/proc/25711/stat : 25711 (python) R 25709 25711 25311 0 -1 4202496 12678 934 1 0 112 11 0 0 25 0 1 0 668241029 294486016 9280 33554432000 4194304 5683644 140737118563152 18446744073709551615 4540565 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/25711/statm: 71896 9280 1826 364 0 7820 0
Current children cumulated CPU time (s) 1.23
Current children cumulated vsize (KiB) 287584

[startup+3.10037 s]
/proc/loadavg: 9.29 8.82 6.78 4/352 25719
/proc/meminfo: memFree=8125248/32950928 swapFree=64721852/67111528
[pid=25711] ppid=25709 vsize=365064 CPUtime=2.04 cores=0-7
/proc/25711/stat : 25711 (python) S 25709 25711 25311 0 -1 4202496 19143 934 1 0 191 13 0 0 25 0 9 0 668241029 373825536 8232 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101524939 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/25711/statm: 91266 8232 1891 364 0 27163 0
[pid=25717] ppid=25711 vsize=283512 CPUtime=0 cores=0-7
/proc/25717/stat : 25717 (python) S 25711 25711 25311 0 -1 4202560 733 0 0 0 0 0 0 0 25 0 1 0 668241238 290316288 6582 33554432000 4194304 5683644 140737118563152 18446744073709551615 248089718287 0 0 16781312 514 15025213700824891391 0 0 17 2 0 0 0
/proc/25717/statm: 70878 6582 355 364 0 6796 0
[pid=25719] ppid=25717 vsize=1018996 CPUtime=1 cores=0-7
/proc/25719/stat : 25719 (gnovelty+2) R 25717 25719 25719 0 -1 4202496 959 0 0 0 100 0 0 0 25 0 1 0 668241238 1043451904 698 33554432000 134512640 135663704 4286575328 18446744073709551615 134524548 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/25719/statm: 254749 698 113 282 0 254465 0
[pid=25711/tid=25718] ppid=25709 vsize=365064 CPUtime=0 cores=0-7
/proc/25711/task/25718/stat : 25718 (python) S 25709 25711 25311 0 -1 4202560 26 934 0 0 0 0 0 0 25 0 9 0 668241238 373825536 8232 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101521841 0 0 16781312 2 18446604453814428096 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3.04
Current children cumulated vsize (KiB) 1667572

[startup+6.3018 s]
/proc/loadavg: 9.26 8.82 6.79 10/373 25740
/proc/meminfo: memFree=8056776/32950928 swapFree=64721852/67111528
[pid=25711] ppid=25709 vsize=365064 CPUtime=2.04 cores=0-7
/proc/25711/stat : 25711 (python) S 25709 25711 25311 0 -1 4202496 19143 934 1 0 191 13 0 0 25 0 9 0 668241029 373825536 8232 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101524939 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/25711/statm: 91266 8232 1891 364 0 27163 0
[pid=25717] ppid=25711 vsize=283512 CPUtime=0 cores=0-7
/proc/25717/stat : 25717 (python) S 25711 25711 25311 0 -1 4202560 900 0 0 0 0 0 0 0 18 0 1 0 668241238 290316288 6605 33554432000 4194304 5683644 140737118563152 18446744073709551615 248089718287 0 0 16781312 514 15025213700824858886 0 0 17 2 0 0 0
/proc/25717/statm: 70878 6605 378 364 0 6796 0
[pid=25719] ppid=25717 vsize=1018996 CPUtime=4.2 cores=0-7
/proc/25719/stat : 25719 (gnovelty+2) R 25717 25719 25719 0 -1 4202496 959 0 0 0 420 0 0 0 25 0 1 0 668241238 1043451904 698 33554432000 134512640 135663704 4286575328 18446744073709551615 134546752 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/25719/statm: 254749 698 113 282 0 254465 0
[pid=25720] ppid=25711 vsize=293700 CPUtime=0.01 cores=0-7
/proc/25720/stat : 25720 (python) S 25711 25711 25311 0 -1 4202560 910 0 0 0 0 1 0 0 18 0 1 0 668241241 300748800 6686 33554432000 4194304 5683644 140737118563152 18446744073709551615 248089718342 0 0 16781312 514 15962043237422563586 0 0 17 1 0 0 0
/proc/25720/statm: 73425 6686 372 364 0 9340 0
[pid=25722] ppid=25720 vsize=1018996 CPUtime=2.21 cores=0-7
/proc/25722/stat : 25722 (gnovelty+2) R 25720 25722 25722 0 -1 4202496 963 0 0 0 221 0 0 0 25 0 1 0 668241242 1043451904 697 33554432000 134512640 135663704 4289226000 18446744073709551615 134524918 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/25722/statm: 254749 697 113 282 0 254465 0
[pid=25723] ppid=25711 vsize=303956 CPUtime=0.01 cores=0-7
/proc/25723/stat : 25723 (python) S 25711 25711 25311 0 -1 4202560 902 0 0 0 0 1 0 0 18 0 1 0 668241244 311250944 6690 33554432000 4194304 5683644 140737118563152 18446744073709551615 248089718342 0 0 16781312 514 3551695515773010180 0 0 17 6 0 0 0
/proc/25723/statm: 75989 6690 372 364 0 11901 0
[pid=25725] ppid=25723 vsize=421428 CPUtime=4.09 cores=0-7
/proc/25725/stat : 25725 (adaptg2wsat2009) R 25723 25725 25725 0 -1 4202496 2053 0 0 0 409 0 0 0 25 0 1 0 668241245 431542272 1782 33554432000 134512640 135107411 4289730544 18446744073709551615 134531561 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/25725/statm: 105357 1782 59 146 0 104233 0
[pid=25726] ppid=25711 vsize=314212 CPUtime=0.01 cores=0-7
/proc/25726/stat : 25726 (python) S 25711 25711 25311 0 -1 4202560 937 0 0 0 0 1 0 0 18 0 1 0 668241248 321753088 6694 33554432000 4194304 5683644 140737118563152 18446744073709551615 248089718342 0 0 16781312 514 15025209852534161668 0 0 17 0 0 0 0
/proc/25726/statm: 78553 6694 372 364 0 14462 0
[pid=25728] ppid=25726 vsize=421428 CPUtime=4.08 cores=0-7
/proc/25728/stat : 25728 (adaptg2wsat2009) R 25726 25728 25728 0 -1 4202496 2046 0 0 0 408 0 0 0 25 0 1 0 668241248 431542272 1782 33554432000 134512640 135107411 4288427952 18446744073709551615 134532277 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/25728/statm: 105357 1782 59 146 0 104233 0
[pid=25729] ppid=25711 vsize=324468 CPUtime=0.01 cores=0-7
/proc/25729/stat : 25729 (python) S 25711 25711 25311 0 -1 4202560 940 0 0 0 1 0 0 0 18 0 1 0 668241251 332255232 6698 33554432000 4194304 5683644 140737118563152 18446744073709551615 248089718342 0 0 16781312 514 10848141813991047426 0 0 17 6 0 0 0
/proc/25729/statm: 81117 6698 372 364 0 17023 0
[pid=25731] ppid=25729 vsize=421428 CPUtime=4.05 cores=0-7
/proc/25731/stat : 25731 (adaptg2wsat2009) R 25729 25731 25731 0 -1 4202496 2047 0 0 0 405 0 0 0 25 0 1 0 668241251 431542272 1782 33554432000 134512640 135107411 4289312448 18446744073709551615 134531600 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/25731/statm: 105357 1782 59 146 0 104233 0
[pid=25732] ppid=25711 vsize=334724 CPUtime=0 cores=0-7
/proc/25732/stat : 25732 (python) S 25711 25711 25311 0 -1 4202560 755 0 0 0 0 0 0 0 18 0 1 0 668241254 342757376 6679 33554432000 4194304 5683644 140737118563152 18446744073709551615 248089718342 0 0 16781312 514 15498760439646355714 0 0 17 0 0 0 0
/proc/25732/statm: 83681 6679 349 364 0 19584 0
[pid=25734] ppid=25732 vsize=1018996 CPUtime=4 cores=0-7
/proc/25734/stat : 25734 (gnovelty+2) R 25732 25734 25734 0 -1 4202496 963 0 0 0 400 0 0 0 25 0 1 0 668241254 1043451904 698 33554432000 134512640 135663704 4290823888 18446744073709551615 134524548 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/25734/statm: 254749 698 113 282 0 254465 0
[pid=25735] ppid=25711 vsize=344980 CPUtime=0 cores=0-7
/proc/25735/stat : 25735 (python) S 25711 25711 25311 0 -1 4202560 762 0 0 0 0 0 0 0 18 0 1 0 668241257 353259520 6683 33554432000 4194304 5683644 140737118563152 18446744073709551615 248089718342 0 0 16781312 514 11367186893725729024 0 0 17 6 0 0 0
/proc/25735/statm: 86245 6683 349 364 0 22145 0
[pid=25737] ppid=25735 vsize=251292 CPUtime=1.95 cores=0-7
/proc/25737/stat : 25737 (FH) R 25735 25737 25737 0 -1 4202496 1951 0 0 0 195 0 0 0 25 0 1 0 668241260 257323008 1683 33554432000 134512640 134936988 4291505920 18446744073709551615 134552504 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/25737/statm: 62823 1683 41 104 0 62327 0
[pid=25738] ppid=25711 vsize=355236 CPUtime=0 cores=0-7
/proc/25738/stat : 25738 (python) S 25711 25711 25311 0 -1 4202560 794 0 0 0 0 0 0 0 18 0 1 0 668241261 363761664 6693 33554432000 4194304 5683644 140737118563152 18446744073709551615 248089718342 0 0 16781312 514 16759271081176432902 0 0 17 6 0 0 0
/proc/25738/statm: 88809 6693 355 364 0 24706 0
[pid=25740] ppid=25738 vsize=421428 CPUtime=3.94 cores=0-7
/proc/25740/stat : 25740 (adaptg2wsat2009) R 25738 25740 25740 0 -1 4202496 2057 0 0 0 394 0 0 0 25 0 1 0 668241261 431542272 1783 33554432000 134512640 135107411 4290941968 18446744073709551615 134531781 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/25740/statm: 105357 1783 59 146 0 104233 0
[pid=25711/tid=25718] ppid=25709 vsize=365064 CPUtime=0 cores=0-7
/proc/25711/task/25718/stat : 25718 (python) S 25709 25711 25311 0 -1 4202560 26 934 0 0 0 0 0 0 25 0 9 0 668241238 373825536 8232 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101521841 0 0 16781312 2 18446604453814428096 0 0 -1 4 0 0 0
[pid=25711/tid=25721] ppid=25709 vsize=365064 CPUtime=0 cores=0-7
/proc/25711/task/25721/stat : 25721 (python) S 25709 25711 25311 0 -1 4202560 24 934 0 0 0 0 0 0 25 0 9 0 668241241 373825536 8232 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101521841 0 0 16781312 2 18446604453814428096 0 0 -1 0 0 0 0
[pid=25711/tid=25724] ppid=25709 vsize=365064 CPUtime=0 cores=0-7
/proc/25711/task/25724/stat : 25724 (python) S 25709 25711 25311 0 -1 4202560 23 934 0 0 0 0 0 0 20 0 9 0 668241244 373825536 8232 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101521841 0 0 16781312 2 18446604453814428096 0 0 -1 0 0 0 0
[pid=25711/tid=25727] ppid=25709 vsize=365064 CPUtime=0 cores=0-7
/proc/25711/task/25727/stat : 25727 (python) S 25709 25711 25311 0 -1 4202560 20 934 0 0 0 0 0 0 22 0 9 0 668241248 373825536 8232 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101521841 0 0 16781312 2 18446604453814428096 0 0 -1 0 0 0 0
[pid=25711/tid=25730] ppid=25709 vsize=365064 CPUtime=0 cores=0-7
/proc/25711/task/25730/stat : 25730 (python) S 25709 25711 25311 0 -1 4202560 27 934 0 0 0 0 0 0 20 0 9 0 668241251 373825536 8232 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101521841 0 0 16781312 2 18446604453814428096 0 0 -1 6 0 0 0
[pid=25711/tid=25733] ppid=25709 vsize=365064 CPUtime=0 cores=0-7
/proc/25711/task/25733/stat : 25733 (python) S 25709 25711 25311 0 -1 4202560 24 934 0 0 0 0 0 0 22 0 9 0 668241254 373825536 8232 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101521841 0 0 16781312 2 18446604453814428096 0 0 -1 0 0 0 0
[pid=25711/tid=25736] ppid=25709 vsize=365064 CPUtime=0 cores=0-7
/proc/25711/task/25736/stat : 25736 (python) S 25709 25711 25311 0 -1 4202560 28 934 0 0 0 0 0 0 24 0 9 0 668241258 373825536 8232 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101521841 0 0 16781312 2 18446604453814428096 0 0 -1 6 0 0 0
[pid=25711/tid=25739] ppid=25709 vsize=365064 CPUtime=0 cores=0-7
/proc/25711/task/25739/stat : 25739 (python) S 25709 25711 25311 0 -1 4202560 29 934 0 0 0 0 0 0 25 0 9 0 668241261 373825536 8232 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101521841 0 0 16781312 2 18446604453814428096 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 30.6
Current children cumulated vsize (KiB) 7913844

[startup+12.7017 s]
/proc/loadavg: 9.24 8.82 6.80 10/373 25741
/proc/meminfo: memFree=8054284/32950928 swapFree=64721852/67111528
[pid=25711] ppid=25709 vsize=365064 CPUtime=2.04 cores=0-7
/proc/25711/stat : 25711 (python) S 25709 25711 25311 0 -1 4202496 19143 934 1 0 191 13 0 0 25 0 9 0 668241029 373825536 8232 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101524939 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/25711/statm: 91266 8232 1891 364 0 27163 0
[pid=25717] ppid=25711 vsize=283512 CPUtime=0.02 cores=0-7
/proc/25717/stat : 25717 (python) S 25711 25711 25311 0 -1 4202560 902 0 0 0 1 1 0 0 18 0 1 0 668241238 290316288 6605 33554432000 4194304 5683644 140737118563152 18446744073709551615 248089718287 0 0 16781312 514 14784578704843440390 0 0 17 2 0 0 0
/proc/25717/statm: 70878 6605 378 364 0 6796 0
[pid=25719] ppid=25717 vsize=1018996 CPUtime=10.6 cores=0-7
/proc/25719/stat : 25719 (gnovelty+2) R 25717 25719 25719 0 -1 4202496 959 0 0 0 1060 0 0 0 25 0 1 0 668241238 1043451904 698 33554432000 134512640 135663704 4286575328 18446744073709551615 134524548 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/25719/statm: 254749 698 113 282 0 254465 0
[pid=25720] ppid=25711 vsize=293700 CPUtime=0.02 cores=0-7
/proc/25720/stat : 25720 (python) S 25711 25711 25311 0 -1 4202560 913 0 0 0 1 1 0 0 18 0 1 0 668241241 300748800 6686 33554432000 4194304 5683644 140737118563152 18446744073709551615 248089718342 0 0 16781312 514 14784578704843440386 0 0 17 1 0 0 0
/proc/25720/statm: 73425 6686 372 364 0 9340 0
[pid=25722] ppid=25720 vsize=1018996 CPUtime=5.41 cores=0-7
/proc/25722/stat : 25722 (gnovelty+2) R 25720 25722 25722 0 -1 4202496 963 0 0 0 541 0 0 0 25 0 1 0 668241242 1043451904 697 33554432000 134512640 135663704 4289226000 18446744073709551615 134545640 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/25722/statm: 254749 697 113 282 0 254465 0
[pid=25723] ppid=25711 vsize=303956 CPUtime=0.02 cores=0-7
/proc/25723/stat : 25723 (python) S 25711 25711 25311 0 -1 4202560 905 0 0 0 1 1 0 0 18 0 1 0 668241244 311250944 6690 33554432000 4194304 5683644 140737118563152 18446744073709551615 248089718342 0 0 16781312 514 14784578704843440388 0 0 17 6 0 0 0
/proc/25723/statm: 75989 6690 372 364 0 11901 0
[pid=25725] ppid=25723 vsize=421428 CPUtime=10.41 cores=0-7
/proc/25725/stat : 25725 (adaptg2wsat2009) R 25723 25725 25725 0 -1 4202496 2053 0 0 0 1041 0 0 0 25 0 1 0 668241245 431542272 1782 33554432000 134512640 135107411 4289730544 18446744073709551615 134531561 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/25725/statm: 105357 1782 59 146 0 104233 0
[pid=25726] ppid=25711 vsize=314212 CPUtime=0.02 cores=0-7
/proc/25726/stat : 25726 (python) S 25711 25711 25311 0 -1 4202560 939 0 0 0 1 1 0 0 18 0 1 0 668241248 321753088 6694 33554432000 4194304 5683644 140737118563152 18446744073709551615 248089718342 0 0 16781312 514 14784578704843440388 0 0 17 0 0 0 0
/proc/25726/statm: 78553 6694 372 364 0 14462 0
[pid=25728] ppid=25726 vsize=421428 CPUtime=10.47 cores=0-7
/proc/25728/stat : 25728 (adaptg2wsat2009) R 25726 25728 25728 0 -1 4202496 2046 0 0 0 1047 0 0 0 25 0 1 0 668241248 431542272 1782 33554432000 134512640 135107411 4288427952 18446744073709551615 134538035 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/25728/statm: 105357 1782 59 146 0 104233 0
[pid=25729] ppid=25711 vsize=324468 CPUtime=0.02 cores=0-7
/proc/25729/stat : 25729 (python) S 25711 25711 25311 0 -1 4202560 944 0 0 0 1 1 0 0 18 0 1 0 668241251 332255232 6698 33554432000 4194304 5683644 140737118563152 18446744073709551615 248089718342 0 0 16781312 514 14784578704843440386 0 0 17 6 0 0 0
/proc/25729/statm: 81117 6698 372 364 0 17023 0
[pid=25731] ppid=25729 vsize=421428 CPUtime=10.44 cores=0-7
/proc/25731/stat : 25731 (adaptg2wsat2009) R 25729 25731 25731 0 -1 4202496 2047 0 0 0 1044 0 0 0 25 0 1 0 668241251 431542272 1782 33554432000 134512640 135107411 4289312448 18446744073709551615 134531810 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/25731/statm: 105357 1782 59 146 0 104233 0
[pid=25732] ppid=25711 vsize=334724 CPUtime=0.02 cores=0-7

################
# More data... #
################

/proc/25815/statm: 127521 7085 408 364 0 63373 0
[pid=25817] ppid=25815 vsize=335996 CPUtime=101.32 cores=0-7
/proc/25817/stat : 25817 (iPAWS) T 25815 25817 25817 0 -1 4202496 1834 0 0 0 10132 0 0 0 25 0 1 0 668293843 344059904 1547 33554432000 134512640 135114423 4294667312 18446744073709551615 134516646 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/25817/statm: 83999 1547 59 147 0 83851 0
[pid=25815/tid=25827] ppid=25711 vsize=510084 CPUtime=0 cores=0-7
/proc/25815/task/25827/stat : 25827 (python) S 25711 25711 25311 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 668306278 522326016 7085 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101521841 0 0 16781312 514 18446604443636376832 0 0 -1 1 0 0 0
[pid=25820] ppid=25711 vsize=520532 CPUtime=0.94 cores=0-7
/proc/25820/stat : 25820 (python) S 25711 25711 25311 0 -1 4202560 1314 0 0 0 47 47 0 0 18 0 2 0 668297965 533024768 7089 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101524939 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/25820/statm: 130133 7089 408 364 0 65982 0
[pid=25822] ppid=25820 vsize=268420 CPUtime=296.76 cores=0-7
/proc/25822/stat : 25822 (hybridGM3) T 25820 25822 25822 0 -1 4202496 5829 0 0 0 29676 0 0 0 25 0 1 0 668297966 274862080 5543 33554432000 134512640 135204385 4292478160 18446744073709551615 134538395 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/25822/statm: 67105 5543 63 169 0 66935 0
[pid=25820/tid=25837] ppid=25711 vsize=520532 CPUtime=0 cores=0-7
/proc/25820/task/25837/stat : 25837 (python) S 25711 25711 25311 0 -1 4202560 49 0 0 0 0 0 0 0 18 0 2 0 668319440 533024768 7089 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101521841 0 0 16781312 514 18446604453828441984 0 0 -1 1 0 0 0
[pid=25828] ppid=25711 vsize=595964 CPUtime=1.09 cores=0-7
/proc/25828/stat : 25828 (python) S 25711 25711 25311 0 -1 4202560 1297 0 0 0 60 49 0 0 18 0 2 0 668306284 610267136 7100 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101524939 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/25828/statm: 148991 7100 402 364 0 84837 0
[pid=25830] ppid=25828 vsize=268424 CPUtime=292.66 cores=0-7
/proc/25830/stat : 25830 (hybridGM3) T 25828 25830 25830 0 -1 4202496 5832 0 0 0 29266 0 0 0 25 0 1 0 668306284 274866176 5544 33554432000 134512640 135204385 4288118480 18446744073709551615 134537726 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/25830/statm: 67106 5544 63 169 0 66936 0
[pid=25828/tid=25848] ppid=25711 vsize=595964 CPUtime=0 cores=0-7
/proc/25828/task/25848/stat : 25848 (python) S 25711 25711 25311 0 -1 4202560 51 0 0 0 0 0 0 0 18 0 2 0 668333177 610267136 7100 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101521841 0 0 16781312 514 18446604443751454720 0 0 -1 2 0 0 0
[pid=25832] ppid=25711 vsize=540888 CPUtime=1.96 cores=0-7
/proc/25832/stat : 25832 (python) S 25711 25711 25311 0 -1 4202560 1122 0 0 0 101 95 0 0 18 0 1 0 668310505 553869312 7067 33554432000 4194304 5683644 140737118563152 18446744073709551615 248089718342 0 0 16781312 514 14784578704843440388 0 0 17 2 0 0 0
/proc/25832/statm: 135222 7067 374 364 0 71065 0
[pid=25834] ppid=25832 vsize=459768 CPUtime=550.08 cores=0-7
/proc/25834/stat : 25834 (TNM) R 25832 25834 25834 0 -1 4202496 1937 0 0 0 55007 1 0 0 25 0 1 0 668310506 470802432 1652 33554432000 134512640 135008748 4289668768 18446744073709551615 134527524 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/25834/statm: 114942 1652 46 122 0 113842 0
[pid=25841] ppid=25711 vsize=616704 CPUtime=0.32 cores=0-7
/proc/25841/stat : 25841 (python) S 25711 25711 25311 0 -1 4202560 1274 0 0 0 16 16 0 0 19 0 2 0 668329581 631504896 7114 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101524939 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/25841/statm: 154176 7114 401 364 0 90016 0
[pid=25843] ppid=25841 vsize=335996 CPUtime=102.83 cores=0-7
/proc/25843/stat : 25843 (iPAWS) T 25841 25843 25843 0 -1 4202496 1831 0 0 0 10283 0 0 0 25 0 1 0 668329582 344059904 1545 33554432000 134512640 135114423 4291760400 18446744073709551615 134514698 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/25843/statm: 83999 1545 59 147 0 83851 0
[pid=25841/tid=25851] ppid=25711 vsize=616704 CPUtime=0 cores=0-7
/proc/25841/task/25851/stat : 25851 (python) S 25711 25711 25311 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 668340212 631504896 7114 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101521841 0 0 16781312 514 18446604453870412096 0 0 -1 0 0 0 0
[pid=25845] ppid=25711 vsize=561520 CPUtime=0.3 cores=0-7
/proc/25845/stat : 25845 (python) S 25711 25711 25311 0 -1 4202560 1271 0 0 0 14 16 0 0 19 0 2 0 668332339 574996480 7125 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101524939 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/25845/statm: 140380 7125 402 364 0 76217 0
[pid=25847] ppid=25845 vsize=1018996 CPUtime=100.31 cores=0-7
/proc/25847/stat : 25847 (gnovelty+2) T 25845 25847 25847 0 -1 4202496 983 0 0 0 10031 0 0 0 25 0 1 0 668332340 1043451904 698 33554432000 134512640 135663704 4290625888 18446744073709551615 134525260 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/25847/statm: 254749 698 113 282 0 254465 0
[pid=25845/tid=25855] ppid=25711 vsize=561520 CPUtime=0 cores=0-7
/proc/25845/task/25855/stat : 25855 (python) S 25711 25711 25311 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 668342373 574996480 7125 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101521841 0 0 16781312 514 18446604453871520704 0 0 -1 2 0 0 0
[pid=25852] ppid=25711 vsize=571816 CPUtime=1.08 cores=0-7
/proc/25852/stat : 25852 (python) S 25711 25711 25311 0 -1 4202560 1357 0 0 0 56 52 0 0 18 0 2 0 668340218 585539584 7161 33554432000 4194304 5683644 140737118563152 18446744073709551615 248089718342 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/25852/statm: 142954 7161 408 364 0 78788 0
[pid=25854] ppid=25852 vsize=268424 CPUtime=340.73 cores=0-7
/proc/25854/stat : 25854 (hybridGM3) R 25852 25854 25854 0 -1 4202496 5849 0 0 0 34073 0 0 0 25 0 1 0 668340219 274866176 5544 33554432000 134512640 135204385 4294325472 18446744073709551615 134538487 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/25854/statm: 67106 5544 63 169 0 66936 0
[pid=25852/tid=25870] ppid=25711 vsize=571816 CPUtime=0 cores=0-7
/proc/25852/task/25870/stat : 25870 (python) S 25711 25711 25311 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 668360484 585539584 7161 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101521841 0 0 16781312 514 18446604449216145408 0 0 -1 1 0 0 0
[pid=25856] ppid=25711 vsize=582132 CPUtime=1.01 cores=0-7
/proc/25856/stat : 25856 (python) S 25711 25711 25311 0 -1 4202560 1328 0 0 0 52 49 0 0 18 0 2 0 668342379 596103168 7176 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101524939 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/25856/statm: 145533 7176 407 364 0 81364 0
[pid=25858] ppid=25856 vsize=268424 CPUtime=292.17 cores=0-7
/proc/25858/stat : 25858 (hybridGM3) T 25856 25858 25858 0 -1 4202496 5835 0 0 0 29217 0 0 0 25 0 1 0 668342379 274866176 5544 33554432000 134512640 135204385 4288646960 18446744073709551615 134545435 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/25858/statm: 67106 5544 63 169 0 66936 0
[pid=25856/tid=25872] ppid=25711 vsize=582132 CPUtime=0 cores=0-7
/proc/25856/task/25872/stat : 25872 (python) S 25711 25711 25311 0 -1 4202560 46 0 0 0 0 0 0 0 18 0 2 0 668362644 596103168 7176 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101521841 0 0 16781312 514 18446604449216144512 0 0 -1 2 0 0 0
[pid=25859] ppid=25711 vsize=592632 CPUtime=0.64 cores=0-7
/proc/25859/stat : 25859 (python) S 25711 25711 25311 0 -1 4202560 1315 0 0 0 32 32 0 0 19 0 2 0 668342716 606855168 7180 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101524939 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/25859/statm: 148158 7180 407 364 0 83986 0
[pid=25861] ppid=25859 vsize=268424 CPUtime=194.87 cores=0-7
/proc/25861/stat : 25861 (hybridGM3) T 25859 25861 25861 0 -1 4202496 5837 0 0 0 19487 0 0 0 25 0 1 0 668342717 274866176 5544 33554432000 134512640 135204385 4290137952 18446744073709551615 134538056 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/25861/statm: 67106 5544 63 169 0 66936 0
[pid=25859/tid=25873] ppid=25711 vsize=592632 CPUtime=0 cores=0-7
/proc/25859/task/25873/stat : 25873 (python) S 25711 25711 25311 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 668364185 606855168 7180 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101521841 0 0 16781312 514 18446604469869512512 0 0 -1 4 0 0 0
[pid=25865] ppid=25711 vsize=602952 CPUtime=0.72 cores=0-7
/proc/25865/stat : 25865 (python) S 25711 25711 25311 0 -1 4202560 1179 0 0 0 39 33 0 0 18 0 1 0 668352495 617422848 7156 33554432000 4194304 5683644 140737118563152 18446744073709551615 248089718342 0 0 16781312 514 14784578704843440389 0 0 17 0 0 0 0
/proc/25865/statm: 150738 7156 380 364 0 86563 0
[pid=25867] ppid=25865 vsize=421364 CPUtime=176.45 cores=0-7
/proc/25867/stat : 25867 (NCVWr) R 25865 25867 25867 0 -1 4202496 2002 0 0 0 17645 0 0 0 25 0 1 0 668352495 431476736 1710 33554432000 134512640 135113558 4287929760 18446744073709551615 134531124 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/25867/statm: 105341 1710 58 147 0 104216 0
[pid=25877] ppid=25711 vsize=613128 CPUtime=0.02 cores=0-7
/proc/25877/stat : 25877 (python) S 25711 25711 25311 0 -1 4202560 1167 0 0 0 1 1 0 0 18 0 1 0 668375697 627843072 7194 33554432000 4194304 5683644 140737118563152 18446744073709551615 248089718342 0 0 16781312 514 4283275918278885638 0 0 17 1 0 0 0
/proc/25877/statm: 153282 7194 380 364 0 89104 0
[pid=25879] ppid=25877 vsize=251292 CPUtime=4.69 cores=0-7
/proc/25879/stat : 25879 (FH) R 25877 25879 25879 0 -1 4202496 1975 0 0 0 469 0 0 0 25 0 1 0 668375697 257323008 1682 33554432000 134512640 134936988 4291792304 18446744073709551615 134558745 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/25879/statm: 62823 1682 41 104 0 62327 0
[pid=25880] ppid=25711 vsize=689180 CPUtime=0 cores=0-7
/proc/25880/stat : 25880 (python) S 25711 25711 25311 0 -1 4202560 944 0 0 0 0 0 0 0 19 0 1 0 668376295 705720320 7167 33554432000 4194304 5683644 140737118563152 18446744073709551615 248089718342 0 0 16781312 514 4283279766569615359 0 0 17 1 0 0 0
/proc/25880/statm: 172295 7167 347 364 0 108114 0
[pid=25882] ppid=25880 vsize=335996 CPUtime=0.15 cores=0-7
/proc/25882/stat : 25882 (iPAWS) R 25880 25882 25882 0 -1 4202496 1839 0 0 0 15 0 0 0 21 0 1 0 668376295 344059904 1547 33554432000 134512640 135114423 4294276480 18446744073709551615 134547183 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/25882/statm: 83999 1547 59 147 0 83851 0
[pid=25711/tid=25718] ppid=25709 vsize=698760 CPUtime=0 cores=0-7
/proc/25711/task/25718/stat : 25718 (python) S 25709 25711 25311 0 -1 4202560 26 934 0 0 0 0 0 0 25 0 35 0 668241238 715530240 8767 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101521841 0 0 16781312 2 18446604453814428096 0 0 -1 4 0 0 0
[pid=25711/tid=25721] ppid=25709 vsize=698760 CPUtime=0 cores=0-7
/proc/25711/task/25721/stat : 25721 (python) S 25709 25711 25311 0 -1 4202560 24 934 0 0 0 0 0 0 25 0 35 0 668241241 715530240 8767 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101521841 0 0 16781312 2 18446604453814428096 0 0 -1 0 0 0 0
[pid=25711/tid=25724] ppid=25709 vsize=698760 CPUtime=0 cores=0-7
/proc/25711/task/25724/stat : 25724 (python) S 25709 25711 25311 0 -1 4202560 27 934 0 0 0 0 0 0 15 0 35 0 668241244 715530240 8767 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101521841 0 0 16781312 2 18446604453814428096 0 0 -1 0 0 0 0
[pid=25711/tid=25727] ppid=25709 vsize=698760 CPUtime=0 cores=0-7
/proc/25711/task/25727/stat : 25727 (python) S 25709 25711 25311 0 -1 4202560 25 934 0 0 0 0 0 0 18 0 35 0 668241248 715530240 8767 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101521841 0 0 16781312 2 18446604453814428096 0 0 -1 0 0 0 0
[pid=25711/tid=25730] ppid=25709 vsize=698760 CPUtime=0 cores=0-7
/proc/25711/task/25730/stat : 25730 (python) S 25709 25711 25311 0 -1 4202560 33 934 0 0 0 0 0 0 18 0 35 0 668241251 715530240 8767 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101521841 0 0 16781312 2 18446604453814428096 0 0 -1 6 0 0 0
[pid=25711/tid=25733] ppid=25709 vsize=698760 CPUtime=0 cores=0-7
/proc/25711/task/25733/stat : 25733 (python) S 25709 25711 25311 0 -1 4202560 24 934 0 0 0 0 0 0 22 0 35 0 668241254 715530240 8767 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101521841 0 0 16781312 2 18446604453814428096 0 0 -1 0 0 0 0
[pid=25711/tid=25736] ppid=25709 vsize=698760 CPUtime=0 cores=0-7
/proc/25711/task/25736/stat : 25736 (python) S 25709 25711 25311 0 -1 4202560 34 934 0 0 0 0 0 0 18 0 35 0 668241258 715530240 8767 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101521841 0 0 16781312 2 18446604453814428096 0 0 -1 6 0 0 0
[pid=25711/tid=25739] ppid=25709 vsize=698760 CPUtime=0 cores=0-7
/proc/25711/task/25739/stat : 25739 (python) S 25709 25711 25311 0 -1 4202560 44 934 0 0 0 0 0 0 18 0 35 0 668241261 715530240 8767 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101521841 0 0 16781312 2 18446604453814428096 0 0 -1 6 0 0 0
[pid=25711/tid=25746] ppid=25709 vsize=698760 CPUtime=0 cores=0-7
/proc/25711/task/25746/stat : 25746 (python) S 25709 25711 25311 0 -1 4202560 32 934 0 0 0 0 0 0 18 0 35 0 668251272 715530240 8767 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101521841 0 0 16781312 2 18446604453814428096 0 0 -1 3 0 0 0
[pid=25711/tid=25750] ppid=25709 vsize=698760 CPUtime=0 cores=0-7
/proc/25711/task/25750/stat : 25750 (python) S 25709 25711 25311 0 -1 4202560 39 934 0 0 0 0 0 0 18 0 35 0 668251281 715530240 8767 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101521841 0 0 16781312 2 18446604453814428096 0 0 -1 3 0 0 0
[pid=25711/tid=25754] ppid=25709 vsize=698760 CPUtime=0 cores=0-7
/proc/25711/task/25754/stat : 25754 (python) S 25709 25711 25311 0 -1 4202560 35 934 0 0 0 0 0 0 18 0 35 0 668251295 715530240 8767 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101521841 0 0 16781312 2 18446604453814428096 0 0 -1 6 0 0 0
[pid=25711/tid=25758] ppid=25709 vsize=698760 CPUtime=0 cores=0-7
/proc/25711/task/25758/stat : 25758 (python) S 25709 25711 25311 0 -1 4202560 27 934 0 0 0 0 0 0 20 0 35 0 668251892 715530240 8767 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101521841 0 0 16781312 2 18446604453814428096 0 0 -1 6 0 0 0
[pid=25711/tid=25762] ppid=25709 vsize=698760 CPUtime=0 cores=0-7
/proc/25711/task/25762/stat : 25762 (python) S 25709 25711 25311 0 -1 4202560 30 934 0 0 0 0 0 0 18 0 35 0 668252479 715530240 8767 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101521841 0 0 16781312 2 18446604453814428096 0 0 -1 6 0 0 0
[pid=25711/tid=25767] ppid=25709 vsize=698760 CPUtime=0 cores=0-7
/proc/25711/task/25767/stat : 25767 (python) S 25709 25711 25311 0 -1 4202560 37 934 0 0 0 0 0 0 18 0 35 0 668253684 715530240 8767 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101521841 0 0 16781312 2 18446604453814428096 0 0 -1 6 0 0 0
[pid=25711/tid=25771] ppid=25709 vsize=698760 CPUtime=0 cores=0-7
/proc/25711/task/25771/stat : 25771 (python) S 25709 25711 25311 0 -1 4202560 28 934 0 0 0 0 0 0 20 0 35 0 668254284 715530240 8767 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101521841 0 0 16781312 2 18446604453814428096 0 0 -1 6 0 0 0
[pid=25711/tid=25775] ppid=25709 vsize=698760 CPUtime=0 cores=0-7
/proc/25711/task/25775/stat : 25775 (python) S 25709 25711 25311 0 -1 4202560 32 934 0 0 0 0 0 0 18 0 35 0 668255508 715530240 8767 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101521841 0 0 16781312 2 18446604453814428096 0 0 -1 5 0 0 0
[pid=25711/tid=25781] ppid=25709 vsize=698760 CPUtime=0 cores=0-7
/proc/25711/task/25781/stat : 25781 (python) S 25709 25711 25311 0 -1 4202560 29 934 0 0 0 0 0 0 20 0 35 0 668262533 715530240 8767 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101521841 0 0 16781312 2 18446604453814428096 0 0 -1 7 0 0 0
[pid=25711/tid=25785] ppid=25709 vsize=698760 CPUtime=0 cores=0-7
/proc/25711/task/25785/stat : 25785 (python) S 25709 25711 25311 0 -1 4202560 31 934 0 0 0 0 0 0 18 0 35 0 668263109 715530240 8767 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101521841 0 0 16781312 2 18446604453814428096 0 0 -1 7 0 0 0
[pid=25711/tid=25794] ppid=25709 vsize=698760 CPUtime=0 cores=0-7
/proc/25711/task/25794/stat : 25794 (python) S 25709 25711 25311 0 -1 4202560 31 934 0 0 0 0 0 0 19 0 35 0 668273456 715530240 8767 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101521841 0 0 16781312 2 18446604453814428096 0 0 -1 2 0 0 0
[pid=25711/tid=25799] ppid=25709 vsize=698760 CPUtime=0 cores=0-7
/proc/25711/task/25799/stat : 25799 (python) S 25709 25711 25311 0 -1 4202560 33 934 0 0 0 0 0 0 18 0 35 0 668274973 715530240 8767 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101521841 0 0 16781312 2 18446604453814428096 0 0 -1 2 0 0 0
[pid=25711/tid=25803] ppid=25709 vsize=698760 CPUtime=0 cores=0-7
/proc/25711/task/25803/stat : 25803 (python) S 25709 25711 25311 0 -1 4202560 29 934 0 0 0 0 0 0 20 0 35 0 668275878 715530240 8767 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101521841 0 0 16781312 2 18446604453814428096 0 0 -1 2 0 0 0
[pid=25711/tid=25810] ppid=25709 vsize=698760 CPUtime=0 cores=0-7
/proc/25711/task/25810/stat : 25810 (python) S 25709 25711 25311 0 -1 4202560 23 934 0 0 0 0 0 0 20 0 35 0 668284102 715530240 8767 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101521841 0 0 16781312 2 18446604453814428096 0 0 -1 1 0 0 0
[pid=25711/tid=25816] ppid=25709 vsize=698760 CPUtime=0 cores=0-7
/proc/25711/task/25816/stat : 25816 (python) S 25709 25711 25311 0 -1 4202560 26 934 0 0 0 0 0 0 19 0 35 0 668293843 715530240 8767 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101521841 0 0 16781312 2 18446604453814428096 0 0 -1 2 0 0 0
[pid=25711/tid=25821] ppid=25709 vsize=698760 CPUtime=0 cores=0-7
/proc/25711/task/25821/stat : 25821 (python) S 25709 25711 25311 0 -1 4202560 31 934 0 0 0 0 0 0 18 0 35 0 668297965 715530240 8767 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101521841 0 0 16781312 2 18446604453814428096 0 0 -1 2 0 0 0
[pid=25711/tid=25829] ppid=25709 vsize=698760 CPUtime=0 cores=0-7
/proc/25711/task/25829/stat : 25829 (python) S 25709 25711 25311 0 -1 4202560 35 934 0 0 0 0 0 0 18 0 35 0 668306284 715530240 8767 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101521841 0 0 16781312 2 18446604453814428096 0 0 -1 2 0 0 0
[pid=25711/tid=25833] ppid=25709 vsize=698760 CPUtime=0 cores=0-7
/proc/25711/task/25833/stat : 25833 (python) S 25709 25711 25311 0 -1 4202560 27 934 0 0 0 0 0 0 20 0 35 0 668310506 715530240 8767 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101521841 0 0 16781312 2 18446604453814428096 0 0 -1 2 0 0 0
[pid=25711/tid=25842] ppid=25709 vsize=698760 CPUtime=0 cores=0-7
/proc/25711/task/25842/stat : 25842 (python) S 25709 25711 25311 0 -1 4202560 25 934 0 0 0 0 0 0 19 0 35 0 668329581 715530240 8767 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101521841 0 0 16781312 2 18446604453814428096 0 0 -1 2 0 0 0
[pid=25711/tid=25846] ppid=25709 vsize=698760 CPUtime=0 cores=0-7
/proc/25711/task/25846/stat : 25846 (python) S 25709 25711 25311 0 -1 4202560 27 934 0 0 0 0 0 0 20 0 35 0 668332339 715530240 8767 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101521841 0 0 16781312 2 18446604453814428096 0 0 -1 2 0 0 0
[pid=25711/tid=25853] ppid=25709 vsize=698760 CPUtime=0 cores=0-7
/proc/25711/task/25853/stat : 25853 (python) S 25709 25711 25311 0 -1 4202560 35 934 0 0 0 0 0 0 18 0 35 0 668340219 715530240 8767 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101521841 0 0 16781312 2 18446604453814428096 0 0 -1 2 0 0 0
[pid=25711/tid=25857] ppid=25709 vsize=698760 CPUtime=0 cores=0-7
/proc/25711/task/25857/stat : 25857 (python) S 25709 25711 25311 0 -1 4202560 30 934 0 0 0 0 0 0 18 0 35 0 668342379 715530240 8767 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101521841 0 0 16781312 2 18446604453814428096 0 0 -1 2 0 0 0
[pid=25711/tid=25860] ppid=25709 vsize=698760 CPUtime=0 cores=0-7
/proc/25711/task/25860/stat : 25860 (python) S 25709 25711 25311 0 -1 4202560 26 934 0 0 0 0 0 0 20 0 35 0 668342717 715530240 8767 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101521841 0 0 16781312 2 18446604453814428096 0 0 -1 2 0 0 0
[pid=25711/tid=25866] ppid=25709 vsize=698760 CPUtime=0 cores=0-7
/proc/25711/task/25866/stat : 25866 (python) S 25709 25711 25311 0 -1 4202560 27 934 0 0 0 0 0 0 20 0 35 0 668352495 715530240 8767 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101521841 0 0 16781312 2 18446604453814428096 0 0 -1 2 0 0 0
[pid=25711/tid=25878] ppid=25709 vsize=698760 CPUtime=0 cores=0-7
/proc/25711/task/25878/stat : 25878 (python) S 25709 25711 25311 0 -1 4202560 26 934 0 0 0 0 0 0 19 0 35 0 668375697 715530240 8767 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101521841 0 0 16781312 2 18446604453814428096 0 0 -1 3 0 0 0
[pid=25711/tid=25881] ppid=25709 vsize=698760 CPUtime=0 cores=0-7
/proc/25711/task/25881/stat : 25881 (python) S 25709 25711 25311 0 -1 4202560 28 934 0 0 0 0 0 0 19 0 35 0 668376295 715530240 8767 33554432000 4194304 5683644 140737118563152 18446744073709551615 248101521841 0 0 16781312 2 18446604453814428096 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 9590.3
Current children cumulated vsize (KiB) 32250672

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 25711 and gives
#  childrusage.ru_utime.tv_sec=1961
#  childrusage.ru_utime.tv_usec=882748
#  childrusage.ru_stime.tv_sec=15
#  childrusage.ru_stime.tv_usec=778601
# CPU time returned by wait4() is 1977.66
# while last known CPU time is 9590.3
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1353.12
CPU time (s): 9590.3
CPU user time (s): 9574.69
CPU system time (s): 15.61
CPU usage (%): 708.756
Max. virtual memory (cumulated for all children) (KiB): 32250672

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1961.88
system time used= 15.7786
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 107747
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 16829
involuntary context switches= 19088

runsolver used 15.2567 second user time and 35.4226 second system time

The end

Launcher Data 

Begin job on node123 at 2011-05-16 19:20:12
IDJOB=3385418
IDBENCH=82683
IDSOLVER=1870
FILE ID=node123/3385418-1305566412
RUNJOBID= node123-1305564964-25325
PBS_JOBID= 13350127
Free space on /tmp= 73280 MiB

SOLVER NAME= borg-sat 11.03.02-par-random (fixed)
BENCH NAME= SAT11/random/large/unif-k5-r20-v1500-c30000-S1305346115-067-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3385418-1305566412/watcher-3385418-1305566412 -o /tmp/evaluation-result-3385418-1305566412/solver-3385418-1305566412 -C 40000 -W 5100 -M 31000  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3385418-1305566412.cnf -seed 1374732334 -budget 40000 -cores 8

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 0c1f90cdbd54dc48a7f7e4333f537217
RANDOM SEED=1374732334

node123.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:       8155656 kB
Buffers:        339312 kB
Cached:        4843912 kB
SwapCached:      10524 kB
Active:       21080548 kB
Inactive:      3398768 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:       8155656 kB
SwapTotal:    67111528 kB
SwapFree:     64721852 kB
Dirty:           32612 kB
Writeback:           0 kB
AnonPages:    19294416 kB
Mapped:          26188 kB
Slab:           182120 kB
PageTables:      63668 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS: 82578840 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73248 MiB
End job on node123 at 2011-05-16 19:42:47