Trace number 3190793

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ppfolio par? (TO) 4800.31 1201.71

General information on the benchmark

Namerandom/large/
unif-k3-r4.2-v25000-c105000-S92008434-079-UNKNOWN.cnf
MD5SUMaeb1e99b5983291023a10a1b4c71c10e
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark40.0889
Satisfiable
(Un)Satisfiability was proved
Number of variables25000
Number of clauses105000
Sum of the clauses size315000
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 3105000
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.00/0.00	This is pico-portfolio, a naive parallel portfolio
0.00/0.00	Copyright (C) 2011 Olivier Roussel
0.00/0.00	Using 4 processing units
0.00/0.00	This portfolio uses the following solvers:
0.00/0.00	[1] cryptominisat 2.7.1 (Mate Soos)
0.00/0.00	[2] clasp 1.3.6 (Martin Gebser, Benjamin Kaufmann, and Torsten Schaub)
0.00/0.00	[3] TNM 2009 (Wanxia Wei and Chu Min Li)
0.00/0.00	[4] lingeling 276-6264d55-100731 (Armin Biere)
0.00/0.00	[5] march_hi 2009 (Marijn Heule and Hans Van Maaren)
0.00/0.00	Starting cryptominisat 2.7.1 on core(s) 1
0.00/0.00	Starting clasp 1.3.6 on core(s) 5
0.00/0.00	Starting TNM 2009 on core(s) 7
0.00/0.00	Starting lingeling 276-6264d55-100731 on core(s) 3
0.00/0.00	Starting march_hi 2009 on core(s) 7

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3190793-1304036256/watcher-3190793-1304036256 -o /tmp/evaluation-result-3190793-1304036256/solver-3190793-1304036256 -C 4800 -W 1300 -M 15500 ppfolio HOME/instance-3190793-1304036256.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 5.15 7.17 8.66 1/171 28753
/proc/meminfo: memFree=15156128/32950928 swapFree=67111528/67111528
[pid=28753] ppid=28751 vsize=18324 CPUtime=0 cores=1,3,5,7
/proc/28753/stat : 28753 (ppfolio) R 28751 28753 27798 0 -1 4202496 518 0 0 0 0 0 0 0 25 0 1 0 515225252 18763776 389 33554432000 4194304 4287625 140734170343872 18446744073709551615 274473987599 0 0 4096 65536 0 0 0 17 1 0 0 0
/proc/28753/statm: 4581 389 341 23 0 91 0

[startup+0.236871 s]
/proc/loadavg: 5.15 7.17 8.66 1/171 28753
/proc/meminfo: memFree=15156128/32950928 swapFree=67111528/67111528
[pid=28753] ppid=28751 vsize=18324 CPUtime=0 cores=1,3,5,7
/proc/28753/stat : 28753 (ppfolio) S 28751 28753 27798 0 -1 4202496 520 0 0 0 0 0 0 0 24 0 1 0 515225252 18763776 389 33554432000 4194304 4287625 140734170343872 18446744073709551615 274473987599 0 0 4096 65536 4608256371077414911 0 0 17 1 0 0 0
/proc/28753/statm: 4581 389 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.300864 s]
/proc/loadavg: 5.15 7.17 8.66 1/171 28753
/proc/meminfo: memFree=15156128/32950928 swapFree=67111528/67111528
[pid=28753] ppid=28751 vsize=18324 CPUtime=0 cores=1,3,5,7
/proc/28753/stat : 28753 (ppfolio) S 28751 28753 27798 0 -1 4202496 520 0 0 0 0 0 0 0 24 0 1 0 515225252 18763776 389 33554432000 4194304 4287625 140734170343872 18446744073709551615 274473987599 0 0 4096 65536 4608256371077414911 0 0 17 1 0 0 0
/proc/28753/statm: 4581 389 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.400845 s]
/proc/loadavg: 5.15 7.17 8.66 1/171 28753
/proc/meminfo: memFree=15156128/32950928 swapFree=67111528/67111528
[pid=28753] ppid=28751 vsize=18324 CPUtime=0 cores=1,3,5,7
/proc/28753/stat : 28753 (ppfolio) S 28751 28753 27798 0 -1 4202496 520 0 0 0 0 0 0 0 24 0 1 0 515225252 18763776 389 33554432000 4194304 4287625 140734170343872 18446744073709551615 274473987599 0 0 4096 65536 4608256371077414911 0 0 17 1 0 0 0
/proc/28753/statm: 4581 389 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.70079 s]
/proc/loadavg: 5.15 7.17 8.66 1/171 28753
/proc/meminfo: memFree=15156128/32950928 swapFree=67111528/67111528
[pid=28753] ppid=28751 vsize=18324 CPUtime=0 cores=1,3,5,7
/proc/28753/stat : 28753 (ppfolio) S 28751 28753 27798 0 -1 4202496 520 0 0 0 0 0 0 0 22 0 1 0 515225252 18763776 389 33554432000 4194304 4287625 140734170343872 18446744073709551615 274473987599 0 0 4096 65536 4608256371077414911 0 0 17 1 0 0 0
/proc/28753/statm: 4581 389 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+1.50066 s]
/proc/loadavg: 5.13 7.13 8.64 6/177 28759
/proc/meminfo: memFree=15076140/32950928 swapFree=67111528/67111528
[pid=28753] ppid=28751 vsize=18324 CPUtime=0 cores=1,3,5,7
/proc/28753/stat : 28753 (ppfolio) S 28751 28753 27798 0 -1 4202496 520 0 0 0 0 0 0 0 22 0 1 0 515225252 18763776 389 33554432000 4194304 4287625 140734170343872 18446744073709551615 274473987599 0 0 4096 65536 4608256371077414911 0 0 17 1 0 0 0
/proc/28753/statm: 4581 389 341 23 0 91 0
[pid=28754] ppid=28753 vsize=47204 CPUtime=1.48 cores=1
/proc/28754/stat : 28754 (cryptominisat) R 28753 28753 27798 0 -1 4202496 9453 0 0 0 147 1 0 0 25 0 1 0 515225253 48336896 7431 33554432000 4194304 4587354 140734486570080 18446744073709551615 4230299 0 0 4096 2 0 0 0 17 1 0 0 0
/proc/28754/statm: 11801 7431 326 96 0 8363 0
[pid=28755] ppid=28753 vsize=26244 CPUtime=1.48 cores=5
/proc/28755/stat : 28755 (clasp) R 28753 28753 27798 0 -1 4202496 5222 0 0 0 148 0 0 0 25 0 1 0 515225253 26873856 3876 33554432000 4194304 4785740 140733447912224 18446744073709551615 4552550 0 0 4096 16386 0 0 0 17 5 0 0 0
/proc/28755/statm: 6561 3876 340 145 0 3606 0
[pid=28756] ppid=28753 vsize=461896 CPUtime=0.71 cores=7
/proc/28756/stat : 28756 (TNM) R 28753 28753 27798 0 -1 4202496 4294 0 0 0 71 0 0 0 25 0 1 0 515225253 472981504 4191 33554432000 134512640 134534564 4287196096 18446744073709551615 134530227 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/28756/statm: 115474 4191 95 6 0 114140 0
[pid=28757] ppid=28753 vsize=11700 CPUtime=1.49 cores=3
/proc/28757/stat : 28757 (lingeling) R 28753 28753 27798 0 -1 4202496 3950 0 0 0 149 0 0 0 25 0 1 0 515225253 11980800 2091 33554432000 4194304 4352236 140733744502416 18446744073709551615 4328112 0 0 4096 17506 0 0 0 17 3 0 0 0
/proc/28757/statm: 2925 2091 127 39 0 2005 0
[pid=28758] ppid=28753 vsize=25588 CPUtime=0.75 cores=7
/proc/28758/stat : 28758 (march_hi) R 28753 28753 27798 0 -1 4202496 20652 0 0 0 73 2 0 0 25 0 1 0 515225253 26202112 3922 33554432000 4194304 4331988 140735061839264 18446744073709551615 4252218 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/28758/statm: 6397 3922 143 34 0 4839 0
Current children cumulated CPU time (s) 5.91
Current children cumulated vsize (KiB) 590956

[startup+3.10037 s]
/proc/loadavg: 5.13 7.13 8.64 6/177 28759
/proc/meminfo: memFree=15058532/32950928 swapFree=67111528/67111528
[pid=28753] ppid=28751 vsize=18324 CPUtime=0 cores=1,3,5,7
/proc/28753/stat : 28753 (ppfolio) S 28751 28753 27798 0 -1 4202496 520 0 0 0 0 0 0 0 15 0 1 0 515225252 18763776 389 33554432000 4194304 4287625 140734170343872 18446744073709551615 274473987599 0 0 4096 65536 4608256371077414911 0 0 17 1 0 0 0
/proc/28753/statm: 4581 389 341 23 0 91 0
[pid=28754] ppid=28753 vsize=84708 CPUtime=3.09 cores=1
/proc/28754/stat : 28754 (cryptominisat) R 28753 28753 27798 0 -1 4202496 10555 0 0 0 307 2 0 0 25 0 1 0 515225253 86740992 8529 33554432000 4194304 4587354 140734486570080 18446744073709551615 4237679 0 0 4096 2 0 0 0 17 1 0 0 0
/proc/28754/statm: 21177 8529 326 96 0 17739 0
[pid=28755] ppid=28753 vsize=33340 CPUtime=3.08 cores=5
/proc/28755/stat : 28755 (clasp) R 28753 28753 27798 0 -1 4202496 6982 0 0 0 308 0 0 0 25 0 1 0 515225253 34140160 5636 33554432000 4194304 4785740 140733447912224 18446744073709551615 4287158 0 0 4096 16386 0 0 0 17 5 0 0 0
/proc/28755/statm: 8335 5636 340 145 0 5380 0
[pid=28756] ppid=28753 vsize=461896 CPUtime=1.51 cores=7
/proc/28756/stat : 28756 (TNM) R 28753 28753 27798 0 -1 4202496 4294 0 0 0 151 0 0 0 25 0 1 0 515225253 472981504 4191 33554432000 134512640 134534564 4287196096 18446744073709551615 134530379 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/28756/statm: 115474 4191 95 6 0 114140 0
[pid=28757] ppid=28753 vsize=11700 CPUtime=3.09 cores=3
/proc/28757/stat : 28757 (lingeling) R 28753 28753 27798 0 -1 4202496 3950 0 0 0 309 0 0 0 25 0 1 0 515225253 11980800 2091 33554432000 4194304 4352236 140733744502416 18446744073709551615 4328026 0 0 4096 17506 0 0 0 17 3 0 0 0
/proc/28757/statm: 2925 2091 127 39 0 2005 0
[pid=28758] ppid=28753 vsize=56516 CPUtime=1.54 cores=7
/proc/28758/stat : 28758 (march_hi) R 28753 28753 27798 0 -1 4202496 34433 0 0 0 150 4 0 0 25 0 1 0 515225253 57872384 11401 33554432000 4194304 4331988 140735061839264 18446744073709551615 4229042 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/28758/statm: 14129 11401 173 34 0 12571 0
Current children cumulated CPU time (s) 12.31
Current children cumulated vsize (KiB) 666484

[startup+6.30081 s]
/proc/loadavg: 5.12 7.10 8.62 6/177 28759
/proc/meminfo: memFree=15011692/32950928 swapFree=67111528/67111528
[pid=28753] ppid=28751 vsize=18324 CPUtime=0 cores=1,3,5,7
/proc/28753/stat : 28753 (ppfolio) S 28751 28753 27798 0 -1 4202496 520 0 0 0 0 0 0 0 15 0 1 0 515225252 18763776 389 33554432000 4194304 4287625 140734170343872 18446744073709551615 274473987599 0 0 4096 65536 4608256371077414911 0 0 17 1 0 0 0
/proc/28753/statm: 4581 389 341 23 0 91 0
[pid=28754] ppid=28753 vsize=84708 CPUtime=6.29 cores=1
/proc/28754/stat : 28754 (cryptominisat) R 28753 28753 27798 0 -1 4202496 12594 0 0 0 626 3 0 0 25 0 1 0 515225253 86740992 10558 33554432000 4194304 4587354 140734486570080 18446744073709551615 4237679 0 0 4096 2 0 0 0 17 1 0 0 0
/proc/28754/statm: 21177 10558 326 96 0 17739 0
[pid=28755] ppid=28753 vsize=48992 CPUtime=6.28 cores=5
/proc/28755/stat : 28755 (clasp) R 28753 28753 27798 0 -1 4202496 10909 0 0 0 627 1 0 0 25 0 1 0 515225253 50167808 9563 33554432000 4194304 4785740 140733447912224 18446744073709551615 4287244 0 0 4096 16386 0 0 0 17 5 0 0 0
/proc/28755/statm: 12248 9563 340 145 0 9293 0
[pid=28756] ppid=28753 vsize=461896 CPUtime=3.11 cores=7
/proc/28756/stat : 28756 (TNM) R 28753 28753 27798 0 -1 4202496 4294 0 0 0 311 0 0 0 25 0 1 0 515225253 472981504 4191 33554432000 134512640 134534564 4287196096 18446744073709551615 134530236 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/28756/statm: 115474 4191 95 6 0 114140 0
[pid=28757] ppid=28753 vsize=11700 CPUtime=6.29 cores=3
/proc/28757/stat : 28757 (lingeling) R 28753 28753 27798 0 -1 4202496 3950 0 0 0 629 0 0 0 25 0 1 0 515225253 11980800 2091 33554432000 4194304 4352236 140733744502416 18446744073709551615 4327936 0 0 4096 17506 0 0 0 17 3 0 0 0
/proc/28757/statm: 2925 2091 127 39 0 2005 0
[pid=28758] ppid=28753 vsize=58008 CPUtime=3.13 cores=7
/proc/28758/stat : 28758 (march_hi) R 28753 28753 27798 0 -1 4202496 34777 0 0 0 308 5 0 0 25 0 1 0 515225253 59400192 11745 33554432000 4194304 4331988 140735061839264 18446744073709551615 4228970 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/28758/statm: 14502 11745 173 34 0 12944 0
Current children cumulated CPU time (s) 25.1
Current children cumulated vsize (KiB) 683628

[startup+12.7007 s]
/proc/loadavg: 5.11 7.06 8.60 6/177 28759
/proc/meminfo: memFree=14967688/32950928 swapFree=67111528/67111528
[pid=28753] ppid=28751 vsize=18324 CPUtime=0 cores=1,3,5,7
/proc/28753/stat : 28753 (ppfolio) S 28751 28753 27798 0 -1 4202496 520 0 0 0 0 0 0 0 15 0 1 0 515225252 18763776 389 33554432000 4194304 4287625 140734170343872 18446744073709551615 274473987599 0 0 4096 65536 4608256371077414911 0 0 17 1 0 0 0
/proc/28753/statm: 4581 389 341 23 0 91 0
[pid=28754] ppid=28753 vsize=191288 CPUtime=12.69 cores=1
/proc/28754/stat : 28754 (cryptominisat) R 28753 28753 27798 0 -1 4202496 36096 0 0 0 1262 7 0 0 25 0 1 0 515225253 195878912 28407 33554432000 4194304 4587354 140734486570080 18446744073709551615 4388345 0 0 4096 2 0 0 0 17 1 0 0 0
/proc/28754/statm: 47822 28407 337 96 0 44384 0
[pid=28755] ppid=28753 vsize=72760 CPUtime=12.69 cores=5
/proc/28755/stat : 28755 (clasp) R 28753 28753 27798 0 -1 4202496 16829 0 0 0 1265 4 0 0 25 0 1 0 515225253 74506240 15483 33554432000 4194304 4785740 140733447912224 18446744073709551615 4552550 0 0 4096 16386 0 0 0 17 5 0 0 0
/proc/28755/statm: 18190 15483 340 145 0 15235 0
[pid=28756] ppid=28753 vsize=461896 CPUtime=6.31 cores=7
/proc/28756/stat : 28756 (TNM) R 28753 28753 27798 0 -1 4202496 4294 0 0 0 631 0 0 0 25 0 1 0 515225253 472981504 4191 33554432000 134512640 134534564 4287196096 18446744073709551615 134530151 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/28756/statm: 115474 4191 95 6 0 114140 0
[pid=28757] ppid=28753 vsize=16388 CPUtime=12.69 cores=3
/proc/28757/stat : 28757 (lingeling) R 28753 28753 27798 0 -1 4202496 5122 0 0 0 1269 0 0 0 25 0 1 0 515225253 16781312 3263 33554432000 4194304 4352236 140733744502416 18446744073709551615 4327902 0 0 4096 17506 0 0 0 17 3 0 0 0
/proc/28757/statm: 4097 3263 127 39 0 3177 0
[pid=28758] ppid=28753 vsize=63280 CPUtime=6.3 cores=7
/proc/28758/stat : 28758 (march_hi) R 28753 28753 27798 0 -1 4202496 35959 0 0 0 625 5 0 0 25 0 1 0 515225253 64798720 12927 33554432000 4194304 4331988 140735061839264 18446744073709551615 4297605 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/28758/statm: 15820 12927 173 34 0 14262 0
Current children cumulated CPU time (s) 50.68
Current children cumulated vsize (KiB) 823936

[startup+25.5064 s]
/proc/loadavg: 5.09 6.96 8.54 6/177 28760
/proc/meminfo: memFree=14955976/32950928 swapFree=67111528/67111528
[pid=28753] ppid=28751 vsize=18324 CPUtime=0 cores=1,3,5,7
/proc/28753/stat : 28753 (ppfolio) S 28751 28753 27798 0 -1 4202496 520 0 0 0 0 0 0 0 15 0 1 0 515225252 18763776 389 33554432000 4194304 4287625 140734170343872 18446744073709551615 274473987599 0 0 4096 65536 4608256371077414911 0 0 17 1 0 0 0
/proc/28753/statm: 4581 389 341 23 0 91 0
[pid=28754] ppid=28753 vsize=2044888 CPUtime=25.49 cores=1
/proc/28754/stat : 28754 (cryptominisat) R 28753 28753 27798 0 -1 4202496 54227 0 0 0 2538 11 0 0 25 0 1 0 515225253 2093965312 12513 33554432000 4194304 4587354 140734486570080 18446744073709551615 4250443 0 0 4096 2 0 0 0 17 1 0 0 0
/proc/28754/statm: 511222 12513 337 96 0 507784 0
[pid=28755] ppid=28753 vsize=80300 CPUtime=25.49 cores=5
/proc/28755/stat : 28755 (clasp) R 28753 28753 27798 0 -1 4202496 18678 0 0 0 2544 5 0 0 25 0 1 0 515225253 82227200 17332 33554432000 4194304 4785740 140733447912224 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 5 0 0 0
/proc/28755/statm: 20075 17332 340 145 0 17120 0
[pid=28756] ppid=28753 vsize=461896 CPUtime=12.65 cores=7
/proc/28756/stat : 28756 (TNM) R 28753 28753 27798 0 -1 4202496 4294 0 0 0 1265 0 0 0 25 0 1 0 515225253 472981504 4191 33554432000 134512640 134534564 4287196096 18446744073709551615 134525956 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/28756/statm: 115474 4191 95 6 0 114140 0
[pid=28757] ppid=28753 vsize=17464 CPUtime=25.49 cores=3
/proc/28757/stat : 28757 (lingeling) R 28753 28753 27798 0 -1 4202496 5391 0 0 0 2549 0 0 0 25 0 1 0 515225253 17883136 3532 33554432000 4194304 4352236 140733744502416 18446744073709551615 4327936 0 0 4096 17506 0 0 0 17 3 0 0 0
/proc/28757/statm: 4366 3532 127 39 0 3446 0
[pid=28758] ppid=28753 vsize=64468 CPUtime=12.69 cores=7
/proc/28758/stat : 28758 (march_hi) R 28753 28753 27798 0 -1 4202496 36280 0 0 0 1264 5 0 0 25 0 1 0 515225253 66015232 13248 33554432000 4194304 4331988 140735061839264 18446744073709551615 4284558 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/28758/statm: 16117 13248 173 34 0 14559 0
Current children cumulated CPU time (s) 101.81
Current children cumulated vsize (KiB) 2687340

[startup+51.1009 s]
/proc/loadavg: 5.12 6.82 8.45 6/177 28760
/proc/meminfo: memFree=14808376/32950928 swapFree=67111528/67111528
[pid=28753] ppid=28751 vsize=18324 CPUtime=0 cores=1,3,5,7
/proc/28753/stat : 28753 (ppfolio) S 28751 28753 27798 0 -1 4202496 520 0 0 0 0 0 0 0 15 0 1 0 515225252 18763776 389 33554432000 4194304 4287625 140734170343872 18446744073709551615 274473987599 0 0 4096 65536 4608256371077414911 0 0 17 1 0 0 0
/proc/28753/statm: 4581 389 341 23 0 91 0
[pid=28754] ppid=28753 vsize=236872 CPUtime=51.09 cores=1
/proc/28754/stat : 28754 (cryptominisat) R 28753 28753 27798 0 -1 4202496 105485 0 0 0 5088 21 0 0 25 0 1 0 515225253 242556928 41458 33554432000 4194304 4587354 140734486570080 18446744073709551615 4388350 0 0 4096 2 0 0 0 17 1 0 0 0
/proc/28754/statm: 59218 41458 337 96 0 55780 0
[pid=28755] ppid=28753 vsize=105384 CPUtime=51.09 cores=5
/proc/28755/stat : 28755 (clasp) R 28753 28753 27798 0 -1 4202496 24811 0 0 0 5103 6 0 0 25 0 1 0 515225253 107913216 23465 33554432000 4194304 4785740 140733447912224 18446744073709551615 4550876 0 0 4096 16386 0 0 0 17 5 0 0 0
/proc/28755/statm: 26346 23465 340 145 0 23391 0
[pid=28756] ppid=28753 vsize=461896 CPUtime=25.41 cores=7
/proc/28756/stat : 28756 (TNM) R 28753 28753 27798 0 -1 4202496 4294 0 0 0 2541 0 0 0 25 0 1 0 515225253 472981504 4191 33554432000 134512640 134534564 4287196096 18446744073709551615 134528687 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/28756/statm: 115474 4191 95 6 0 114140 0
[pid=28757] ppid=28753 vsize=23104 CPUtime=51.09 cores=3

################
# More data... #
################

/proc/28758/statm: 17272 14413 173 34 0 15714 0
Current children cumulated CPU time (s) 3764.07
Current children cumulated vsize (KiB) 3001564

[startup+1002.31 s]
/proc/loadavg: 5.11 5.15 6.27 6/177 28786
/proc/meminfo: memFree=14638660/32950928 swapFree=67111528/67111528
[pid=28753] ppid=28751 vsize=18324 CPUtime=0 cores=1,3,5,7
/proc/28753/stat : 28753 (ppfolio) S 28751 28753 27798 0 -1 4202496 520 0 0 0 0 0 0 0 15 0 1 0 515225252 18763776 389 33554432000 4194304 4287625 140734170343872 18446744073709551615 274473987599 0 0 4096 65536 4608256371077414911 0 0 17 1 0 0 0
/proc/28753/statm: 4581 389 341 23 0 91 0
[pid=28754] ppid=28753 vsize=2076772 CPUtime=1002.32 cores=1
/proc/28754/stat : 28754 (cryptominisat) R 28753 28753 27798 0 -1 4202496 614446 0 0 0 99969 263 0 0 25 0 1 0 515225253 2126614528 26155 33554432000 4194304 4587354 140734486570080 18446744073709551615 4238370 0 0 4096 2 0 0 0 17 1 0 0 0
/proc/28754/statm: 519193 26155 337 96 0 515755 0
[pid=28755] ppid=28753 vsize=330524 CPUtime=1002.3 cores=5
/proc/28755/stat : 28755 (clasp) R 28753 28753 27798 0 -1 4202496 80229 0 0 0 100203 27 0 0 25 0 1 0 515225253 338456576 78883 33554432000 4194304 4785740 140733447912224 18446744073709551615 4538886 0 0 4096 16386 0 0 0 17 5 0 0 0
/proc/28755/statm: 82631 78883 340 145 0 79676 0
[pid=28756] ppid=28753 vsize=461896 CPUtime=498.42 cores=7
/proc/28756/stat : 28756 (TNM) R 28753 28753 27798 0 -1 4202496 4294 0 0 0 49842 0 0 0 25 0 1 0 515225253 472981504 4191 33554432000 134512640 134534564 4287196096 18446744073709551615 134528979 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/28756/statm: 115474 4191 95 6 0 114140 0
[pid=28757] ppid=28753 vsize=28976 CPUtime=1002.31 cores=3
/proc/28757/stat : 28757 (lingeling) R 28753 28753 27798 0 -1 4202496 8269 0 0 0 100231 0 0 0 25 0 1 0 515225253 29671424 6410 33554432000 4194304 4352236 140733744502416 18446744073709551615 4328100 0 0 4096 17506 0 0 0 17 3 0 0 0
/proc/28757/statm: 7244 6410 127 39 0 6324 0
[pid=28758] ppid=28753 vsize=69220 CPUtime=498.42 cores=7
/proc/28758/stat : 28758 (march_hi) R 28753 28753 27798 0 -1 4202496 37467 0 0 0 49836 6 0 0 25 0 1 0 515225253 70881280 14435 33554432000 4194304 4331988 140735061839264 18446744073709551615 4263156 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/28758/statm: 17305 14435 173 34 0 15747 0
Current children cumulated CPU time (s) 4003.77
Current children cumulated vsize (KiB) 2985712

[startup+1062.3 s]
/proc/loadavg: 5.17 5.16 6.20 6/177 28787
/proc/meminfo: memFree=14645064/32950928 swapFree=67111528/67111528
[pid=28753] ppid=28751 vsize=18324 CPUtime=0 cores=1,3,5,7
/proc/28753/stat : 28753 (ppfolio) S 28751 28753 27798 0 -1 4202496 520 0 0 0 0 0 0 0 15 0 1 0 515225252 18763776 389 33554432000 4194304 4287625 140734170343872 18446744073709551615 274473987599 0 0 4096 65536 4608256371077414911 0 0 17 1 0 0 0
/proc/28753/statm: 4581 389 341 23 0 91 0
[pid=28754] ppid=28753 vsize=2102080 CPUtime=1062.31 cores=1
/proc/28754/stat : 28754 (cryptominisat) R 28753 28753 27798 0 -1 4202496 642840 0 0 0 105949 282 0 0 25 0 1 0 515225253 2152529920 22125 33554432000 4194304 4587354 140734486570080 18446744073709551615 4241321 0 0 4096 2 0 0 0 17 1 0 0 0
/proc/28754/statm: 525520 22125 337 96 0 522082 0
[pid=28755] ppid=28753 vsize=341012 CPUtime=1062.3 cores=5
/proc/28755/stat : 28755 (clasp) R 28753 28753 27798 0 -1 4202496 82842 0 0 0 106201 29 0 0 25 0 1 0 515225253 349196288 81496 33554432000 4194304 4785740 140733447912224 18446744073709551615 4553402 0 0 4096 16386 0 0 0 17 5 0 0 0
/proc/28755/statm: 85253 81496 340 145 0 82298 0
[pid=28756] ppid=28753 vsize=461896 CPUtime=528.23 cores=7
/proc/28756/stat : 28756 (TNM) R 28753 28753 27798 0 -1 4202496 4294 0 0 0 52823 0 0 0 25 0 1 0 515225253 472981504 4191 33554432000 134512640 134534564 4287196096 18446744073709551615 134530659 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/28756/statm: 115474 4191 95 6 0 114140 0
[pid=28757] ppid=28753 vsize=28976 CPUtime=1062.31 cores=3
/proc/28757/stat : 28757 (lingeling) R 28753 28753 27798 0 -1 4202496 8269 0 0 0 106231 0 0 0 25 0 1 0 515225253 29671424 6410 33554432000 4194304 4352236 140733744502416 18446744073709551615 4328026 0 0 4096 17506 0 0 0 17 3 0 0 0
/proc/28757/statm: 7244 6410 127 39 0 6324 0
[pid=28758] ppid=28753 vsize=69352 CPUtime=528.28 cores=7
/proc/28758/stat : 28758 (march_hi) R 28753 28753 27798 0 -1 4202496 37490 0 0 0 52822 6 0 0 25 0 1 0 515225253 71016448 14458 33554432000 4194304 4331988 140735061839264 18446744073709551615 4226440 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/28758/statm: 17338 14458 173 34 0 15780 0
Current children cumulated CPU time (s) 4243.43
Current children cumulated vsize (KiB) 3021640

[startup+1122.3 s]
/proc/loadavg: 5.12 5.14 6.13 6/177 28789
/proc/meminfo: memFree=14667096/32950928 swapFree=67111528/67111528
[pid=28753] ppid=28751 vsize=18324 CPUtime=0 cores=1,3,5,7
/proc/28753/stat : 28753 (ppfolio) S 28751 28753 27798 0 -1 4202496 520 0 0 0 0 0 0 0 15 0 1 0 515225252 18763776 389 33554432000 4194304 4287625 140734170343872 18446744073709551615 274473987599 0 0 4096 65536 4608256371077414911 0 0 17 1 0 0 0
/proc/28753/statm: 4581 389 341 23 0 91 0
[pid=28754] ppid=28753 vsize=2080852 CPUtime=1122.31 cores=1
/proc/28754/stat : 28754 (cryptominisat) R 28753 28753 27798 0 -1 4202496 670692 0 0 0 111931 300 0 0 25 0 1 0 515225253 2130792448 16390 33554432000 4194304 4587354 140734486570080 18446744073709551615 4238301 0 0 4096 2 0 0 0 17 1 0 0 0
/proc/28754/statm: 520213 16390 337 96 0 516775 0
[pid=28755] ppid=28753 vsize=341012 CPUtime=1122.3 cores=5
/proc/28755/stat : 28755 (clasp) R 28753 28753 27798 0 -1 4202496 82846 0 0 0 112201 29 0 0 25 0 1 0 515225253 349196288 81500 33554432000 4194304 4785740 140733447912224 18446744073709551615 4287204 0 0 4096 16386 0 0 0 17 5 0 0 0
/proc/28755/statm: 85253 81500 340 145 0 82298 0
[pid=28756] ppid=28753 vsize=461896 CPUtime=558.07 cores=7
/proc/28756/stat : 28756 (TNM) R 28753 28753 27798 0 -1 4202496 4294 0 0 0 55807 0 0 0 25 0 1 0 515225253 472981504 4191 33554432000 134512640 134534564 4287196096 18446744073709551615 134526304 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/28756/statm: 115474 4191 95 6 0 114140 0
[pid=28757] ppid=28753 vsize=28976 CPUtime=1122.31 cores=3
/proc/28757/stat : 28757 (lingeling) R 28753 28753 27798 0 -1 4202496 8269 0 0 0 112231 0 0 0 25 0 1 0 515225253 29671424 6410 33554432000 4194304 4352236 140733744502416 18446744073709551615 4271184 0 0 4096 17506 0 0 0 17 3 0 0 0
/proc/28757/statm: 7244 6410 127 39 0 6324 0
[pid=28758] ppid=28753 vsize=69352 CPUtime=558.11 cores=7
/proc/28758/stat : 28758 (march_hi) R 28753 28753 27798 0 -1 4202496 37508 0 0 0 55805 6 0 0 25 0 1 0 515225253 71016448 14476 33554432000 4194304 4331988 140735061839264 18446744073709551615 4228983 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/28758/statm: 17338 14476 173 34 0 15780 0
Current children cumulated CPU time (s) 4483.1
Current children cumulated vsize (KiB) 3000412

[startup+1182.3 s]
/proc/loadavg: 5.16 5.14 6.07 6/177 28791
/proc/meminfo: memFree=14618196/32950928 swapFree=67111528/67111528
[pid=28753] ppid=28751 vsize=18324 CPUtime=0 cores=1,3,5,7
/proc/28753/stat : 28753 (ppfolio) S 28751 28753 27798 0 -1 4202496 520 0 0 0 0 0 0 0 15 0 1 0 515225252 18763776 389 33554432000 4194304 4287625 140734170343872 18446744073709551615 274473987599 0 0 4096 65536 4608256371077414911 0 0 17 1 0 0 0
/proc/28753/statm: 4581 389 341 23 0 91 0
[pid=28754] ppid=28753 vsize=2097868 CPUtime=1182.32 cores=1
/proc/28754/stat : 28754 (cryptominisat) R 28753 28753 27798 0 -1 4202496 697501 0 0 0 117915 317 0 0 25 0 1 0 515225253 2148216832 26160 33554432000 4194304 4587354 140734486570080 18446744073709551615 4238658 0 0 4096 2 0 0 0 17 1 0 0 0
/proc/28754/statm: 524467 26160 337 96 0 521029 0
[pid=28755] ppid=28753 vsize=351812 CPUtime=1182.3 cores=5
/proc/28755/stat : 28755 (clasp) R 28753 28753 27798 0 -1 4202496 85503 0 0 0 118200 30 0 0 25 0 1 0 515225253 360255488 84157 33554432000 4194304 4785740 140733447912224 18446744073709551615 4553003 0 0 4096 16386 0 0 0 17 5 0 0 0
/proc/28755/statm: 87953 84157 340 145 0 84998 0
[pid=28756] ppid=28753 vsize=461896 CPUtime=587.93 cores=7
/proc/28756/stat : 28756 (TNM) R 28753 28753 27798 0 -1 4202496 4294 0 0 0 58793 0 0 0 25 0 1 0 515225253 472981504 4191 33554432000 134512640 134534564 4287196096 18446744073709551615 134528979 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/28756/statm: 115474 4191 95 6 0 114140 0
[pid=28757] ppid=28753 vsize=28976 CPUtime=1182.31 cores=3
/proc/28757/stat : 28757 (lingeling) R 28753 28753 27798 0 -1 4202496 8269 0 0 0 118231 0 0 0 25 0 1 0 515225253 29671424 6410 33554432000 4194304 4352236 140733744502416 18446744073709551615 4328105 0 0 4096 17506 0 0 0 17 3 0 0 0
/proc/28757/statm: 7244 6410 127 39 0 6324 0
[pid=28758] ppid=28753 vsize=69484 CPUtime=587.93 cores=7
/proc/28758/stat : 28758 (march_hi) R 28753 28753 27798 0 -1 4202496 37525 0 0 0 58787 6 0 0 25 0 1 0 515225253 71151616 14493 33554432000 4194304 4331988 140735061839264 18446744073709551615 4226440 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/28758/statm: 17371 14493 173 34 0 15813 0
Current children cumulated CPU time (s) 4722.79
Current children cumulated vsize (KiB) 3028360



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1201.7 s]
/proc/loadavg: 5.11 5.13 6.04 6/177 28791
/proc/meminfo: memFree=14647816/32950928 swapFree=67111528/67111528
[pid=28753] ppid=28751 vsize=18324 CPUtime=0 cores=1,3,5,7
/proc/28753/stat : 28753 (ppfolio) S 28751 28753 27798 0 -1 4202496 520 0 0 0 0 0 0 0 15 0 1 0 515225252 18763776 389 33554432000 4194304 4287625 140734170343872 18446744073709551615 274473987599 0 0 4096 65536 4608256371077414911 0 0 17 1 0 0 0
/proc/28753/statm: 4581 389 341 23 0 91 0
[pid=28754] ppid=28753 vsize=2084896 CPUtime=1201.72 cores=1
/proc/28754/stat : 28754 (cryptominisat) R 28753 28753 27798 0 -1 4202496 706861 0 0 0 119850 322 0 0 25 0 1 0 515225253 2134933504 18191 33554432000 4194304 4587354 140734486570080 18446744073709551615 4237679 0 0 4096 2 0 0 0 17 1 0 0 0
/proc/28754/statm: 521224 18191 337 96 0 517786 0
[pid=28755] ppid=28753 vsize=352916 CPUtime=1201.71 cores=5
/proc/28755/stat : 28755 (clasp) R 28753 28753 27798 0 -1 4202496 85758 0 0 0 120141 30 0 0 25 0 1 0 515225253 361385984 84412 33554432000 4194304 4785740 140733447912224 18446744073709551615 4286953 0 0 4096 16386 0 0 0 17 5 0 0 0
/proc/28755/statm: 88229 84412 340 145 0 85274 0
[pid=28756] ppid=28753 vsize=461896 CPUtime=597.55 cores=7
/proc/28756/stat : 28756 (TNM) R 28753 28753 27798 0 -1 4202496 4294 0 0 0 59755 0 0 0 25 0 1 0 515225253 472981504 4191 33554432000 134512640 134534564 4287196096 18446744073709551615 134530549 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/28756/statm: 115474 4191 95 6 0 114140 0
[pid=28757] ppid=28753 vsize=28976 CPUtime=1201.72 cores=3
/proc/28757/stat : 28757 (lingeling) R 28753 28753 27798 0 -1 4202496 8269 0 0 0 120172 0 0 0 25 0 1 0 515225253 29671424 6410 33554432000 4194304 4352236 140733744502416 18446744073709551615 4328026 0 0 4096 17506 0 0 0 17 3 0 0 0
/proc/28757/statm: 7244 6410 127 39 0 6324 0
[pid=28758] ppid=28753 vsize=69484 CPUtime=597.61 cores=7
/proc/28758/stat : 28758 (march_hi) R 28753 28753 27798 0 -1 4202496 37531 0 0 0 59755 6 0 0 25 0 1 0 515225253 71151616 14499 33554432000 4194304 4331988 140735061839264 18446744073709551615 4211416 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/28758/statm: 17371 14499 173 34 0 15813 0
Current children cumulated CPU time (s) 4800.31
Current children cumulated vsize (KiB) 3016492

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1201.7 s]
/proc/loadavg: 5.11 5.13 6.04 6/177 28791
/proc/meminfo: memFree=14647816/32950928 swapFree=67111528/67111528
[pid=28753] ppid=28751 vsize=18324 CPUtime=0 cores=1,3,5,7
/proc/28753/stat : 28753 (ppfolio) S 28751 28753 27798 0 -1 4202496 520 0 0 0 0 0 0 0 15 0 1 0 515225252 18763776 389 33554432000 4194304 4287625 140734170343872 18446744073709551615 274473987599 0 0 4096 65536 4608256371077414911 0 0 17 1 0 0 0
/proc/28753/statm: 4581 389 341 23 0 91 0
[pid=28754] ppid=28753 vsize=2084896 CPUtime=1201.72 cores=1
/proc/28754/stat : 28754 (cryptominisat) R 28753 28753 27798 0 -1 4202496 706861 0 0 0 119850 322 0 0 25 0 1 0 515225253 2134933504 18191 33554432000 4194304 4587354 140734486570080 18446744073709551615 4237679 0 0 4096 2 0 0 0 17 1 0 0 0
/proc/28754/statm: 521224 18191 337 96 0 517786 0
[pid=28755] ppid=28753 vsize=352916 CPUtime=1201.71 cores=5
/proc/28755/stat : 28755 (clasp) R 28753 28753 27798 0 -1 4202496 85758 0 0 0 120141 30 0 0 25 0 1 0 515225253 361385984 84412 33554432000 4194304 4785740 140733447912224 18446744073709551615 4286953 0 0 4096 16386 0 0 0 17 5 0 0 0
/proc/28755/statm: 88229 84412 340 145 0 85274 0
[pid=28756] ppid=28753 vsize=461896 CPUtime=597.55 cores=7
/proc/28756/stat : 28756 (TNM) R 28753 28753 27798 0 -1 4202496 4294 0 0 0 59755 0 0 0 25 0 1 0 515225253 472981504 4191 33554432000 134512640 134534564 4287196096 18446744073709551615 134530549 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/28756/statm: 115474 4191 95 6 0 114140 0
[pid=28757] ppid=28753 vsize=28976 CPUtime=1201.72 cores=3
/proc/28757/stat : 28757 (lingeling) R 28753 28753 27798 0 -1 4202496 8269 0 0 0 120172 0 0 0 25 0 1 0 515225253 29671424 6410 33554432000 4194304 4352236 140733744502416 18446744073709551615 4328026 0 0 4096 17506 0 0 0 17 3 0 0 0
/proc/28757/statm: 7244 6410 127 39 0 6324 0
[pid=28758] ppid=28753 vsize=69484 CPUtime=597.61 cores=7
/proc/28758/stat : 28758 (march_hi) R 28753 28753 27798 0 -1 4202496 37531 0 0 0 59755 6 0 0 25 0 1 0 515225253 71151616 14499 33554432000 4194304 4331988 140735061839264 18446744073709551615 4211416 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/28758/statm: 17371 14499 173 34 0 15813 0
Current children cumulated CPU time (s) 4800.31
Current children cumulated vsize (KiB) 3016492

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 28753 and gives
#  childrusage.ru_utime.tv_sec=0
#  childrusage.ru_utime.tv_usec=1999
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=1999
# CPU time returned by wait4() is 0.003998
# while last known CPU time is 4800.31
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1201.71
CPU time (s): 4800.31
CPU user time (s): 4796.73
CPU system time (s): 3.58
CPU usage (%): 399.456
Max. virtual memory (cumulated for all children) (KiB): 5012592

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 0.001999
system time used= 0.001999
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 520
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 255
involuntary context switches= 6

runsolver used 1.83272 second user time and 4.73128 second system time

The end

Launcher Data 

Begin job on node125 at 2011-04-29 02:17:36
IDJOB=3190793
IDBENCH=82524
IDSOLVER=1612
FILE ID=node125/3190793-1304036256
RUNJOBID= node125-1304032406-28176
PBS_JOBID= 13170327
Free space on /tmp= 72760 MiB

SOLVER NAME= ppfolio par
BENCH NAME= SAT11/random/large/unif-k3-r4.2-v25000-c105000-S92008434-079-UNKNOWN.cnf
COMMAND LINE= ppfolio BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3190793-1304036256/watcher-3190793-1304036256 -o /tmp/evaluation-result-3190793-1304036256/solver-3190793-1304036256 -C 4800 -W 1300 -M 15500  ppfolio HOME/instance-3190793-1304036256.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= aeb1e99b5983291023a10a1b4c71c10e
RANDOM SEED=1798552184

node125.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5266.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.84
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      15156284 kB
Buffers:       2217476 kB
Cached:       14977792 kB
SwapCached:          0 kB
Active:        9042492 kB
Inactive:      8221108 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      15156284 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:           12040 kB
Writeback:           0 kB
AnonPages:       67336 kB
Mapped:          14712 kB
Slab:           468436 kB
PageTables:       4024 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:   189344 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72752 MiB
End job on node125 at 2011-04-29 02:37:40