Trace number 3258521

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ppfolio par? (TO) 4800.18 1201.71

General information on the benchmark

NameCRAFTED/rbsat/random/
forced/rbsat-v2100c215164gyes7.cnf
MD5SUM140eea74f7e66501965365160d02fa4d
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables2100
Number of clauses215164
Sum of the clauses size432288
Maximum clause length30
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2215094
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 570

Solver Data 

0.00/0.80	This is pico-portfolio, a naive parallel portfolio
0.00/0.80	Copyright (C) 2011 Olivier Roussel
0.00/0.80	Using 4 processing units
0.00/0.80	This portfolio uses the following solvers:
0.00/0.80	[1] cryptominisat 2.7.1 (Mate Soos)
0.00/0.80	[2] clasp 1.3.6 (Martin Gebser, Benjamin Kaufmann, and Torsten Schaub)
0.00/0.80	[3] TNM 2009 (Wanxia Wei and Chu Min Li)
0.00/0.80	[4] lingeling 276-6264d55-100731 (Armin Biere)
0.00/0.80	[5] march_hi 2009 (Marijn Heule and Hans Van Maaren)
0.00/0.80	Starting cryptominisat 2.7.1 on core(s) 0
0.00/0.80	Starting clasp 1.3.6 on core(s) 4
0.00/0.80	Starting TNM 2009 on core(s) 6
0.00/0.80	Starting lingeling 276-6264d55-100731 on core(s) 2
0.00/0.80	Starting march_hi 2009 on core(s) 6

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3258521-1304034019/watcher-3258521-1304034019 -o /tmp/evaluation-result-3258521-1304034019/solver-3258521-1304034019 -C 4800 -W 1300 -M 15500 ppfolio HOME/instance-3258521-1304034019.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 10.08 9.65 8.89 7/182 31734
/proc/meminfo: memFree=16370756/32951124 swapFree=67111528/67111528
[pid=31734] ppid=31732 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/31734/stat : 31734 (ppfolio) R 31732 31734 31124 0 -1 4202496 518 0 0 0 0 0 0 0 25 0 1 0 515002320 18763776 389 33554432000 4194304 4287625 140735753437104 18446744073709551615 230809185807 0 0 4096 65536 0 0 0 17 4 0 0 0
/proc/31734/statm: 4581 389 341 23 0 91 0

[startup+0.267781 s]
/proc/loadavg: 10.08 9.65 8.89 7/182 31734
/proc/meminfo: memFree=16370756/32951124 swapFree=67111528/67111528
[pid=31734] ppid=31732 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/31734/stat : 31734 (ppfolio) R 31732 31734 31124 0 -1 4202496 518 0 0 0 0 0 0 0 25 0 1 0 515002320 18763776 389 33554432000 4194304 4287625 140735753437104 18446744073709551615 230809185807 0 0 4096 65536 0 0 0 17 4 0 0 0
/proc/31734/statm: 4581 389 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.300776 s]
/proc/loadavg: 10.08 9.65 8.89 7/182 31734
/proc/meminfo: memFree=16370756/32951124 swapFree=67111528/67111528
[pid=31734] ppid=31732 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/31734/stat : 31734 (ppfolio) R 31732 31734 31124 0 -1 4202496 518 0 0 0 0 0 0 0 25 0 1 0 515002320 18763776 389 33554432000 4194304 4287625 140735753437104 18446744073709551615 230809185807 0 0 4096 65536 0 0 0 17 4 0 0 0
/proc/31734/statm: 4581 389 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.400758 s]
/proc/loadavg: 10.08 9.65 8.89 7/182 31734
/proc/meminfo: memFree=16370756/32951124 swapFree=67111528/67111528
[pid=31734] ppid=31732 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/31734/stat : 31734 (ppfolio) R 31732 31734 31124 0 -1 4202496 518 0 0 0 0 0 0 0 25 0 1 0 515002320 18763776 389 33554432000 4194304 4287625 140735753437104 18446744073709551615 230809185807 0 0 4096 65536 0 0 0 17 4 0 0 0
/proc/31734/statm: 4581 389 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.700703 s]
/proc/loadavg: 10.08 9.65 8.89 7/182 31734
/proc/meminfo: memFree=16370756/32951124 swapFree=67111528/67111528
[pid=31734] ppid=31732 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/31734/stat : 31734 (ppfolio) R 31732 31734 31124 0 -1 4202496 518 0 0 0 0 0 0 0 25 0 1 0 515002320 18763776 389 33554432000 4194304 4287625 140735753437104 18446744073709551615 230809185807 0 0 4096 65536 0 0 0 17 4 0 0 0
/proc/31734/statm: 4581 389 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+1.50055 s]
/proc/loadavg: 10.08 9.65 8.89 12/188 31740
/proc/meminfo: memFree=16271136/32951124 swapFree=67111528/67111528
[pid=31734] ppid=31732 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/31734/stat : 31734 (ppfolio) S 31732 31734 31124 0 -1 4202496 520 0 0 0 0 0 0 0 17 0 1 0 515002320 18763776 389 33554432000 4194304 4287625 140735753437104 18446744073709551615 230809185807 0 0 4096 65536 3320947607521460223 0 0 17 4 0 0 0
/proc/31734/statm: 4581 389 341 23 0 91 0
[pid=31735] ppid=31734 vsize=68592 CPUtime=1.48 cores=0
/proc/31735/stat : 31735 (cryptominisat) R 31734 31734 31124 0 -1 4202496 13013 0 0 0 147 1 0 0 20 0 1 0 515002320 70238208 11638 33554432000 4194304 4587354 140736710413584 18446744073709551615 4238654 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/31735/statm: 17148 11638 326 96 0 13710 0
[pid=31736] ppid=31734 vsize=26944 CPUtime=1.48 cores=4
/proc/31736/stat : 31736 (clasp) R 31734 31734 31124 0 -1 4202496 5673 0 0 0 148 0 0 0 20 0 1 0 515002320 27590656 3975 33554432000 4194304 4785740 140733237954256 18446744073709551615 4552626 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/31736/statm: 6736 3975 340 145 0 3781 0
[pid=31737] ppid=31734 vsize=461896 CPUtime=0.77 cores=6
/proc/31737/stat : 31737 (TNM) R 31734 31734 31124 0 -1 4202496 3615 0 0 0 76 1 0 0 25 0 1 0 515002320 472981504 3512 33554432000 134512640 134534564 4293057920 18446744073709551615 134524508 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/31737/statm: 115474 3512 91 6 0 114140 0
[pid=31738] ppid=31734 vsize=16176 CPUtime=1.49 cores=2
/proc/31738/stat : 31738 (lingeling) R 31734 31734 31124 0 -1 4202496 4093 0 0 0 149 0 0 0 25 0 1 0 515002320 16564224 2932 33554432000 4194304 4352236 140737136649808 18446744073709551615 4208824 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/31738/statm: 4044 2932 123 39 0 3124 0
[pid=31739] ppid=31734 vsize=36064 CPUtime=0.7 cores=6
/proc/31739/stat : 31739 (march_hi) R 31734 31734 31124 0 -1 4202496 16276 0 0 0 68 2 0 0 25 0 1 0 515002320 36929536 3865 33554432000 4194304 4331988 140733447867200 18446744073709551615 4210459 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/31739/statm: 9016 3865 171 34 0 7458 0
Current children cumulated CPU time (s) 5.92
Current children cumulated vsize (KiB) 627996

[startup+3.10126 s]
/proc/loadavg: 10.16 9.67 8.90 11/188 31740
/proc/meminfo: memFree=16260972/32951124 swapFree=67111528/67111528
[pid=31734] ppid=31732 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/31734/stat : 31734 (ppfolio) S 31732 31734 31124 0 -1 4202496 520 0 0 0 0 0 0 0 15 0 1 0 515002320 18763776 389 33554432000 4194304 4287625 140735753437104 18446744073709551615 230809185807 0 0 4096 65536 3320947607521460223 0 0 17 4 0 0 0
/proc/31734/statm: 4581 389 341 23 0 91 0
[pid=31735] ppid=31734 vsize=1821616 CPUtime=3.09 cores=0
/proc/31735/stat : 31735 (cryptominisat) R 31734 31734 31124 0 -1 4202496 14797 0 0 0 307 2 0 0 25 0 1 0 515002320 1865334784 11994 33554432000 4194304 4587354 140736710413584 18446744073709551615 4230266 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/31735/statm: 455404 11994 337 96 0 451966 0
[pid=31736] ppid=31734 vsize=27208 CPUtime=3.08 cores=4
/proc/31736/stat : 31736 (clasp) R 31734 31734 31124 0 -1 4202496 5856 0 0 0 308 0 0 0 25 0 1 0 515002320 27860992 4158 33554432000 4194304 4785740 140733237954256 18446744073709551615 4286953 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/31736/statm: 6802 4158 340 145 0 3847 0
[pid=31737] ppid=31734 vsize=467476 CPUtime=1.56 cores=6
/proc/31737/stat : 31737 (TNM) R 31734 31734 31124 0 -1 4202496 5161 0 0 0 155 1 0 0 25 0 1 0 515002320 478695424 5058 33554432000 134512640 134534564 4293057920 18446744073709551615 134530360 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/31737/statm: 116869 5058 95 6 0 115535 0
[pid=31738] ppid=31734 vsize=19316 CPUtime=3.09 cores=2
/proc/31738/stat : 31738 (lingeling) R 31734 31734 31124 0 -1 4202496 5151 0 0 0 309 0 0 0 25 0 1 0 515002320 19779584 3990 33554432000 4194304 4352236 140737136649808 18446744073709551615 4231179 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/31738/statm: 4829 3990 126 39 0 3909 0
[pid=31739] ppid=31734 vsize=36640 CPUtime=1.5 cores=6
/proc/31739/stat : 31739 (march_hi) R 31734 31734 31124 0 -1 4202496 16422 0 0 0 148 2 0 0 25 0 1 0 515002320 37519360 4011 33554432000 4194304 4331988 140733447867200 18446744073709551615 4226474 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/31739/statm: 9160 4011 175 34 0 7602 0
Current children cumulated CPU time (s) 12.32
Current children cumulated vsize (KiB) 2390580

[startup+6.30067 s]
/proc/loadavg: 10.16 9.67 8.90 11/188 31740
/proc/meminfo: memFree=16242992/32951124 swapFree=67111528/67111528
[pid=31734] ppid=31732 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/31734/stat : 31734 (ppfolio) S 31732 31734 31124 0 -1 4202496 520 0 0 0 0 0 0 0 15 0 1 0 515002320 18763776 389 33554432000 4194304 4287625 140735753437104 18446744073709551615 230809185807 0 0 4096 65536 3320947607521460223 0 0 17 4 0 0 0
/proc/31734/statm: 4581 389 341 23 0 91 0
[pid=31735] ppid=31734 vsize=1821616 CPUtime=6.29 cores=0
/proc/31735/stat : 31735 (cryptominisat) R 31734 31734 31124 0 -1 4202496 14797 0 0 0 627 2 0 0 25 0 1 0 515002320 1865334784 11994 33554432000 4194304 4587354 140736710413584 18446744073709551615 4365232 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/31735/statm: 455404 11994 337 96 0 451966 0
[pid=31736] ppid=31734 vsize=37908 CPUtime=6.28 cores=4
/proc/31736/stat : 31736 (clasp) R 31734 31734 31124 0 -1 4202496 8512 0 0 0 627 1 0 0 25 0 1 0 515002320 38817792 6814 33554432000 4194304 4785740 140733237954256 18446744073709551615 4550444 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/31736/statm: 9477 6814 340 145 0 6522 0
[pid=31737] ppid=31734 vsize=467476 CPUtime=3.24 cores=6
/proc/31737/stat : 31737 (TNM) R 31734 31734 31124 0 -1 4202496 5161 0 0 0 323 1 0 0 25 0 1 0 515002320 478695424 5058 33554432000 134512640 134534564 4293057920 18446744073709551615 134530360 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/31737/statm: 116869 5058 95 6 0 115535 0
[pid=31738] ppid=31734 vsize=23156 CPUtime=6.28 cores=2
/proc/31738/stat : 31738 (lingeling) R 31734 31734 31124 0 -1 4202496 5911 0 0 0 628 0 0 0 25 0 1 0 515002320 23711744 4750 33554432000 4194304 4352236 140737136649808 18446744073709551615 4328359 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/31738/statm: 5789 4750 126 39 0 4869 0
[pid=31739] ppid=31734 vsize=36916 CPUtime=3.01 cores=6
/proc/31739/stat : 31739 (march_hi) R 31734 31734 31124 0 -1 4202496 16481 0 0 0 299 2 0 0 25 0 1 0 515002320 37801984 4070 33554432000 4194304 4331988 140733447867200 18446744073709551615 4226184 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/31739/statm: 9229 4070 175 34 0 7671 0
Current children cumulated CPU time (s) 25.1
Current children cumulated vsize (KiB) 2405396

[startup+12.7005 s]
/proc/loadavg: 10.13 9.68 8.92 11/188 31741
/proc/meminfo: memFree=16208684/32951124 swapFree=67111528/67111528
[pid=31734] ppid=31732 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/31734/stat : 31734 (ppfolio) S 31732 31734 31124 0 -1 4202496 520 0 0 0 0 0 0 0 15 0 1 0 515002320 18763776 389 33554432000 4194304 4287625 140735753437104 18446744073709551615 230809185807 0 0 4096 65536 3320947607521460223 0 0 17 4 0 0 0
/proc/31734/statm: 4581 389 341 23 0 91 0
[pid=31735] ppid=31734 vsize=1831012 CPUtime=12.69 cores=0
/proc/31735/stat : 31735 (cryptominisat) R 31734 31734 31124 0 -1 4202496 20612 0 0 0 1266 3 0 0 25 0 1 0 515002320 1874956288 15444 33554432000 4194304 4587354 140736710413584 18446744073709551615 4365352 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/31735/statm: 457753 15444 337 96 0 454315 0
[pid=31736] ppid=31734 vsize=53836 CPUtime=12.69 cores=4
/proc/31736/stat : 31736 (clasp) R 31734 31734 31124 0 -1 4202496 12443 0 0 0 1266 3 0 0 25 0 1 0 515002320 55128064 10745 33554432000 4194304 4785740 140733237954256 18446744073709551615 4286953 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/31736/statm: 13459 10745 340 145 0 10504 0
[pid=31737] ppid=31734 vsize=467476 CPUtime=6.41 cores=6
/proc/31737/stat : 31737 (TNM) R 31734 31734 31124 0 -1 4202496 5161 0 0 0 640 1 0 0 25 0 1 0 515002320 478695424 5058 33554432000 134512640 134534564 4293057920 18446744073709551615 134528726 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/31737/statm: 116869 5058 95 6 0 115535 0
[pid=31738] ppid=31734 vsize=25640 CPUtime=12.68 cores=2
/proc/31738/stat : 31738 (lingeling) R 31734 31734 31124 0 -1 4202496 6732 0 0 0 1268 0 0 0 25 0 1 0 515002320 26255360 5571 33554432000 4194304 4352236 140737136649808 18446744073709551615 4328055 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/31738/statm: 6410 5571 126 39 0 5490 0
[pid=31739] ppid=31734 vsize=37236 CPUtime=6.2 cores=6
/proc/31739/stat : 31739 (march_hi) R 31734 31734 31124 0 -1 4202496 16566 0 0 0 618 2 0 0 25 0 1 0 515002320 38129664 4154 33554432000 4194304 4331988 140733447867200 18446744073709551615 4239523 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/31739/statm: 9309 4154 175 34 0 7751 0
Current children cumulated CPU time (s) 50.67
Current children cumulated vsize (KiB) 2433524

[startup+25.5072 s]
/proc/loadavg: 10.19 9.71 8.93 11/188 31741
/proc/meminfo: memFree=16165272/32951124 swapFree=67111528/67111528
[pid=31734] ppid=31732 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/31734/stat : 31734 (ppfolio) S 31732 31734 31124 0 -1 4202496 520 0 0 0 0 0 0 0 15 0 1 0 515002320 18763776 389 33554432000 4194304 4287625 140735753437104 18446744073709551615 230809185807 0 0 4096 65536 3320947607521460223 0 0 17 4 0 0 0
/proc/31734/statm: 4581 389 341 23 0 91 0
[pid=31735] ppid=31734 vsize=1836424 CPUtime=25.5 cores=0
/proc/31735/stat : 31735 (cryptominisat) R 31734 31734 31124 0 -1 4202496 25813 0 0 0 2546 4 0 0 25 0 1 0 515002320 1880498176 20644 33554432000 4194304 4587354 140736710413584 18446744073709551615 4238522 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/31735/statm: 459106 20644 337 96 0 455668 0
[pid=31736] ppid=31734 vsize=58728 CPUtime=25.49 cores=4
/proc/31736/stat : 31736 (clasp) R 31734 31734 31124 0 -1 4202496 13672 0 0 0 2546 3 0 0 25 0 1 0 515002320 60137472 11974 33554432000 4194304 4785740 140733237954256 18446744073709551615 4553411 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/31736/statm: 14682 11974 340 145 0 11727 0
[pid=31737] ppid=31734 vsize=467476 CPUtime=12.75 cores=6
/proc/31737/stat : 31737 (TNM) R 31734 31734 31124 0 -1 4202496 5161 0 0 0 1274 1 0 0 25 0 1 0 515002320 478695424 5058 33554432000 134512640 134534564 4293057920 18446744073709551615 134528708 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/31737/statm: 116869 5058 95 6 0 115535 0
[pid=31738] ppid=31734 vsize=29480 CPUtime=25.49 cores=2
/proc/31738/stat : 31738 (lingeling) R 31734 31734 31124 0 -1 4202496 7692 0 0 0 2549 0 0 0 25 0 1 0 515002320 30187520 6531 33554432000 4194304 4352236 140737136649808 18446744073709551615 4328026 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/31738/statm: 7370 6531 126 39 0 6450 0
[pid=31739] ppid=31734 vsize=37996 CPUtime=12.6 cores=6
/proc/31739/stat : 31739 (march_hi) R 31734 31734 31124 0 -1 4202496 16758 0 0 0 1258 2 0 0 25 0 1 0 515002320 38907904 4345 33554432000 4194304 4331988 140733447867200 18446744073709551615 4228247 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/31739/statm: 9499 4345 175 34 0 7941 0
Current children cumulated CPU time (s) 101.83
Current children cumulated vsize (KiB) 2448428

[startup+51.1005 s]
/proc/loadavg: 10.12 9.73 8.96 11/188 31742
/proc/meminfo: memFree=16095616/32951124 swapFree=67111528/67111528
[pid=31734] ppid=31732 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/31734/stat : 31734 (ppfolio) S 31732 31734 31124 0 -1 4202496 520 0 0 0 0 0 0 0 15 0 1 0 515002320 18763776 389 33554432000 4194304 4287625 140735753437104 18446744073709551615 230809185807 0 0 4096 65536 3320947607521460223 0 0 17 4 0 0 0
/proc/31734/statm: 4581 389 341 23 0 91 0
[pid=31735] ppid=31734 vsize=1881556 CPUtime=51.09 cores=0
/proc/31735/stat : 31735 (cryptominisat) R 31734 31734 31124 0 -1 4202496 43116 0 0 0 5102 7 0 0 25 0 1 0 515002320 1926713344 28532 33554432000 4194304 4587354 140736710413584 18446744073709551615 4238683 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/31735/statm: 470389 28532 337 96 0 466951 0
[pid=31736] ppid=31734 vsize=68664 CPUtime=51.08 cores=4
/proc/31736/stat : 31736 (clasp) R 31734 31734 31124 0 -1 4202496 16044 0 0 0 5104 4 0 0 25 0 1 0 515002320 70311936 14346 33554432000 4194304 4785740 140733237954256 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/31736/statm: 17166 14346 340 145 0 14211 0
[pid=31737] ppid=31734 vsize=467476 CPUtime=25.54 cores=6
/proc/31737/stat : 31737 (TNM) R 31734 31734 31124 0 -1 4202496 5161 0 0 0 2553 1 0 0 25 0 1 0 515002320 478695424 5058 33554432000 134512640 134534564 4293057920 18446744073709551615 134530360 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/31737/statm: 116869 5058 95 6 0 115535 0
[pid=31738] ppid=31734 vsize=36452 CPUtime=51.09 cores=2

################
# More data... #
################

/proc/31739/statm: 18895 11675 175 34 0 17337 0
Current children cumulated CPU time (s) 3764.02
Current children cumulated vsize (KiB) 2758796

[startup+1002.31 s]
/proc/loadavg: 10.19 10.19 9.74 11/188 31767
/proc/meminfo: memFree=15622672/32951124 swapFree=67111528/67111528
[pid=31734] ppid=31732 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/31734/stat : 31734 (ppfolio) S 31732 31734 31124 0 -1 4202496 520 0 0 0 0 0 0 0 15 0 1 0 515002320 18763776 389 33554432000 4194304 4287625 140735753437104 18446744073709551615 230809185807 0 0 4096 65536 3320947607521460223 0 0 17 4 0 0 0
/proc/31734/statm: 4581 389 341 23 0 91 0
[pid=31735] ppid=31734 vsize=1954796 CPUtime=1002.32 cores=0
/proc/31735/stat : 31735 (cryptominisat) R 31734 31734 31124 0 -1 4202496 288152 0 0 0 100124 108 0 0 25 0 1 0 515002320 2001711104 48875 33554432000 4194304 4587354 140736710413584 18446744073709551615 4238578 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/31735/statm: 488699 48875 337 96 0 485261 0
[pid=31736] ppid=31734 vsize=174904 CPUtime=1002.21 cores=4
/proc/31736/stat : 31736 (clasp) R 31734 31734 31124 0 -1 4202496 41288 0 0 0 100209 12 0 0 25 0 1 0 515002320 179101696 39590 33554432000 4194304 4785740 140733237954256 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/31736/statm: 43726 39590 340 145 0 40771 0
[pid=31737] ppid=31734 vsize=467476 CPUtime=498.57 cores=6
/proc/31737/stat : 31737 (TNM) R 31734 31734 31124 0 -1 4202496 5161 0 0 0 49856 1 0 0 25 0 1 0 515002320 478695424 5058 33554432000 134512640 134534564 4293057920 18446744073709551615 134528710 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/31737/statm: 116869 5058 95 6 0 115535 0
[pid=31738] ppid=31734 vsize=77424 CPUtime=1002.32 cores=2
/proc/31738/stat : 31738 (lingeling) R 31734 31734 31124 0 -1 4202496 99550 0 0 0 100220 12 0 0 25 0 1 0 515002320 79282176 18517 33554432000 4194304 4352236 140737136649808 18446744073709551615 4328105 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/31738/statm: 19356 18517 126 39 0 18436 0
[pid=31739] ppid=31734 vsize=75580 CPUtime=498.29 cores=6
/proc/31739/stat : 31739 (march_hi) R 31734 31734 31124 0 -1 4202496 24523 0 0 0 49823 6 0 0 25 0 1 0 515002320 77393920 12110 33554432000 4194304 4331988 140733447867200 18446744073709551615 4228247 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/31739/statm: 18895 12110 175 34 0 17337 0
Current children cumulated CPU time (s) 4003.71
Current children cumulated vsize (KiB) 2768504

[startup+1062.3 s]
/proc/loadavg: 10.13 10.17 9.76 11/188 31769
/proc/meminfo: memFree=15622768/32951124 swapFree=67111528/67111528
[pid=31734] ppid=31732 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/31734/stat : 31734 (ppfolio) S 31732 31734 31124 0 -1 4202496 520 0 0 0 0 0 0 0 15 0 1 0 515002320 18763776 389 33554432000 4194304 4287625 140735753437104 18446744073709551615 230809185807 0 0 4096 65536 3320947607521460223 0 0 17 4 0 0 0
/proc/31734/statm: 4581 389 341 23 0 91 0
[pid=31735] ppid=31734 vsize=1944360 CPUtime=1062.31 cores=0
/proc/31735/stat : 31735 (cryptominisat) R 31734 31734 31124 0 -1 4202496 302543 0 0 0 106116 115 0 0 25 0 1 0 515002320 1991024640 46628 33554432000 4194304 4587354 140736710413584 18446744073709551615 4238690 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/31735/statm: 486090 46628 337 96 0 482652 0
[pid=31736] ppid=31734 vsize=177084 CPUtime=1062.2 cores=4
/proc/31736/stat : 31736 (clasp) R 31734 31734 31124 0 -1 4202496 41779 0 0 0 106208 12 0 0 25 0 1 0 515002320 181334016 40081 33554432000 4194304 4785740 140733237954256 18446744073709551615 4287017 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/31736/statm: 44271 40081 340 145 0 41316 0
[pid=31737] ppid=31734 vsize=467476 CPUtime=528.4 cores=6
/proc/31737/stat : 31737 (TNM) R 31734 31734 31124 0 -1 4202496 5161 0 0 0 52839 1 0 0 25 0 1 0 515002320 478695424 5058 33554432000 134512640 134534564 4293057920 18446744073709551615 134528957 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/31737/statm: 116869 5058 95 6 0 115535 0
[pid=31738] ppid=31734 vsize=77424 CPUtime=1062.32 cores=2
/proc/31738/stat : 31738 (lingeling) R 31734 31734 31124 0 -1 4202496 108766 0 0 0 106218 14 0 0 25 0 1 0 515002320 79282176 18517 33554432000 4194304 4352236 140737136649808 18446744073709551615 4328075 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/31738/statm: 19356 18517 126 39 0 18436 0
[pid=31739] ppid=31734 vsize=84312 CPUtime=528.13 cores=6
/proc/31739/stat : 31739 (march_hi) R 31734 31734 31124 0 -1 4202496 24931 0 0 0 52807 6 0 0 25 0 1 0 515002320 86335488 12518 33554432000 4194304 4331988 140733447867200 18446744073709551615 4237096 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/31739/statm: 21078 12518 175 34 0 19520 0
Current children cumulated CPU time (s) 4243.36
Current children cumulated vsize (KiB) 2768980

[startup+1122.3 s]
/proc/loadavg: 9.68 10.05 9.74 10/187 31803
/proc/meminfo: memFree=14384176/32951124 swapFree=67111528/67111528
[pid=31734] ppid=31732 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/31734/stat : 31734 (ppfolio) S 31732 31734 31124 0 -1 4202496 520 0 0 0 0 0 0 0 15 0 1 0 515002320 18763776 389 33554432000 4194304 4287625 140735753437104 18446744073709551615 230809185807 0 0 4096 65536 3320947607521460223 0 0 17 4 0 0 0
/proc/31734/statm: 4581 389 341 23 0 91 0
[pid=31735] ppid=31734 vsize=1944584 CPUtime=1122.32 cores=0
/proc/31735/stat : 31735 (cryptominisat) R 31734 31734 31124 0 -1 4202496 315977 0 0 0 112112 120 0 0 25 0 1 0 515002320 1991254016 46984 33554432000 4194304 4587354 140736710413584 18446744073709551615 4237600 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/31735/statm: 486146 46984 337 96 0 482708 0
[pid=31736] ppid=31734 vsize=178316 CPUtime=1122.2 cores=4
/proc/31736/stat : 31736 (clasp) R 31734 31734 31124 0 -1 4202496 42042 0 0 0 112208 12 0 0 25 0 1 0 515002320 182595584 40344 33554432000 4194304 4785740 140733237954256 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/31736/statm: 44579 40344 340 145 0 41624 0
[pid=31737] ppid=31734 vsize=467476 CPUtime=558.25 cores=6
/proc/31737/stat : 31737 (TNM) R 31734 31734 31124 0 -1 4202496 5161 0 0 0 55824 1 0 0 25 0 1 0 515002320 478695424 5058 33554432000 134512640 134534564 4293057920 18446744073709551615 134530551 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/31737/statm: 116869 5058 95 6 0 115535 0
[pid=31738] ppid=31734 vsize=75376 CPUtime=1122.28 cores=2
/proc/31738/stat : 31738 (lingeling) R 31734 31734 31124 0 -1 4202496 116446 0 0 0 112210 18 0 0 25 0 1 0 515002320 77185024 18005 33554432000 4194304 4352236 140737136649808 18446744073709551615 4328159 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/31738/statm: 18844 18005 126 39 0 17924 0
[pid=31739] ppid=31734 vsize=84312 CPUtime=557.93 cores=6
/proc/31739/stat : 31739 (march_hi) R 31734 31734 31124 0 -1 4202496 25407 0 0 0 55787 6 0 0 25 0 1 0 515002320 86335488 12994 33554432000 4194304 4331988 140733447867200 18446744073709551615 4228247 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/31739/statm: 21078 12994 175 34 0 19520 0
Current children cumulated CPU time (s) 4482.98
Current children cumulated vsize (KiB) 2768388

[startup+1182.3 s]
/proc/loadavg: 9.37 9.91 9.71 10/187 31805
/proc/meminfo: memFree=14048256/32951124 swapFree=67111528/67111528
[pid=31734] ppid=31732 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/31734/stat : 31734 (ppfolio) S 31732 31734 31124 0 -1 4202496 520 0 0 0 0 0 0 0 15 0 1 0 515002320 18763776 389 33554432000 4194304 4287625 140735753437104 18446744073709551615 230809185807 0 0 4096 65536 3320947607521460223 0 0 17 4 0 0 0
/proc/31734/statm: 4581 389 341 23 0 91 0
[pid=31735] ppid=31734 vsize=1946192 CPUtime=1182.32 cores=0
/proc/31735/stat : 31735 (cryptominisat) R 31734 31734 31124 0 -1 4202496 330824 0 0 0 118103 129 0 0 25 0 1 0 515002320 1992900608 48177 33554432000 4194304 4587354 140736710413584 18446744073709551615 4238611 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/31735/statm: 486548 48177 337 96 0 483110 0
[pid=31736] ppid=31734 vsize=180640 CPUtime=1182.19 cores=4
/proc/31736/stat : 31736 (clasp) R 31734 31734 31124 0 -1 4202496 42611 0 0 0 118207 12 0 0 25 0 1 0 515002320 184975360 40913 33554432000 4194304 4785740 140733237954256 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/31736/statm: 45160 40913 340 145 0 42205 0
[pid=31737] ppid=31734 vsize=467476 CPUtime=588.06 cores=6
/proc/31737/stat : 31737 (TNM) R 31734 31734 31124 0 -1 4202496 5161 0 0 0 58805 1 0 0 25 0 1 0 515002320 478695424 5058 33554432000 134512640 134534564 4293057920 18446744073709551615 134530367 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/31737/statm: 116869 5058 95 6 0 115535 0
[pid=31738] ppid=31734 vsize=77424 CPUtime=1182.29 cores=2
/proc/31738/stat : 31738 (lingeling) R 31734 31734 31124 0 -1 4202496 126174 0 0 0 118209 20 0 0 25 0 1 0 515002320 79282176 18517 33554432000 4194304 4352236 140737136649808 18446744073709551615 4276395 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/31738/statm: 19356 18517 126 39 0 18436 0
[pid=31739] ppid=31734 vsize=84312 CPUtime=587.82 cores=6
/proc/31739/stat : 31739 (march_hi) R 31734 31734 31124 0 -1 4202496 25919 0 0 0 58775 7 0 0 25 0 1 0 515002320 86335488 13506 33554432000 4194304 4331988 140733447867200 18446744073709551615 4239538 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/31739/statm: 21078 13506 175 34 0 19520 0
Current children cumulated CPU time (s) 4722.68
Current children cumulated vsize (KiB) 2774368



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1201.7 s]
/proc/loadavg: 9.27 9.85 9.69 9/186 31805
/proc/meminfo: memFree=13993676/32951124 swapFree=67111528/67111528
[pid=31734] ppid=31732 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/31734/stat : 31734 (ppfolio) S 31732 31734 31124 0 -1 4202496 520 0 0 0 0 0 0 0 15 0 1 0 515002320 18763776 389 33554432000 4194304 4287625 140735753437104 18446744073709551615 230809185807 0 0 4096 65536 3320947607521460223 0 0 17 4 0 0 0
/proc/31734/statm: 4581 389 341 23 0 91 0
[pid=31735] ppid=31734 vsize=1943800 CPUtime=1201.72 cores=0
/proc/31735/stat : 31735 (cryptominisat) R 31734 31734 31124 0 -1 4202496 335487 0 0 0 120041 131 0 0 25 0 1 0 515002320 1990451200 48156 33554432000 4194304 4587354 140736710413584 18446744073709551615 4238618 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/31735/statm: 485950 48156 337 96 0 482512 0
[pid=31736] ppid=31734 vsize=180640 CPUtime=1201.6 cores=4
/proc/31736/stat : 31736 (clasp) R 31734 31734 31124 0 -1 4202496 42623 0 0 0 120148 12 0 0 25 0 1 0 515002320 184975360 40925 33554432000 4194304 4785740 140733237954256 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/31736/statm: 45160 40925 340 145 0 42205 0
[pid=31737] ppid=31734 vsize=467476 CPUtime=597.75 cores=6
/proc/31737/stat : 31737 (TNM) R 31734 31734 31124 0 -1 4202496 5161 0 0 0 59774 1 0 0 25 0 1 0 515002320 478695424 5058 33554432000 134512640 134534564 4293057920 18446744073709551615 134530551 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/31737/statm: 116869 5058 95 6 0 115535 0
[pid=31738] ppid=31734 vsize=77424 CPUtime=1201.69 cores=2
/proc/31738/stat : 31738 (lingeling) R 31734 31734 31124 0 -1 4202496 129246 0 0 0 120146 23 0 0 25 0 1 0 515002320 79282176 18517 33554432000 4194304 4352236 140737136649808 18446744073709551615 4328194 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/31738/statm: 19356 18517 126 39 0 18436 0
[pid=31739] ppid=31734 vsize=84312 CPUtime=597.42 cores=6
/proc/31739/stat : 31739 (march_hi) R 31734 31734 31124 0 -1 4202496 26085 0 0 0 59735 7 0 0 25 0 1 0 515002320 86335488 13672 33554432000 4194304 4331988 140733447867200 18446744073709551615 4228235 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/31739/statm: 21078 13672 175 34 0 19520 0
Current children cumulated CPU time (s) 4800.18
Current children cumulated vsize (KiB) 2771976

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1201.7 s]
/proc/loadavg: 9.27 9.85 9.69 9/186 31805
/proc/meminfo: memFree=13993676/32951124 swapFree=67111528/67111528
[pid=31734] ppid=31732 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/31734/stat : 31734 (ppfolio) S 31732 31734 31124 0 -1 4202496 520 0 0 0 0 0 0 0 15 0 1 0 515002320 18763776 389 33554432000 4194304 4287625 140735753437104 18446744073709551615 230809185807 0 0 4096 65536 3320947607521460223 0 0 17 4 0 0 0
/proc/31734/statm: 4581 389 341 23 0 91 0
[pid=31735] ppid=31734 vsize=1943800 CPUtime=1201.72 cores=0
/proc/31735/stat : 31735 (cryptominisat) R 31734 31734 31124 0 -1 4202496 335487 0 0 0 120041 131 0 0 25 0 1 0 515002320 1990451200 48156 33554432000 4194304 4587354 140736710413584 18446744073709551615 4238618 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/31735/statm: 485950 48156 337 96 0 482512 0
[pid=31736] ppid=31734 vsize=180640 CPUtime=1201.6 cores=4
/proc/31736/stat : 31736 (clasp) R 31734 31734 31124 0 -1 4202496 42623 0 0 0 120148 12 0 0 25 0 1 0 515002320 184975360 40925 33554432000 4194304 4785740 140733237954256 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/31736/statm: 45160 40925 340 145 0 42205 0
[pid=31737] ppid=31734 vsize=467476 CPUtime=597.75 cores=6
/proc/31737/stat : 31737 (TNM) R 31734 31734 31124 0 -1 4202496 5161 0 0 0 59774 1 0 0 25 0 1 0 515002320 478695424 5058 33554432000 134512640 134534564 4293057920 18446744073709551615 134530551 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/31737/statm: 116869 5058 95 6 0 115535 0
[pid=31738] ppid=31734 vsize=77424 CPUtime=1201.69 cores=2
/proc/31738/stat : 31738 (lingeling) R 31734 31734 31124 0 -1 4202496 129246 0 0 0 120146 23 0 0 25 0 1 0 515002320 79282176 18517 33554432000 4194304 4352236 140737136649808 18446744073709551615 4328194 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/31738/statm: 19356 18517 126 39 0 18436 0
[pid=31739] ppid=31734 vsize=84312 CPUtime=597.42 cores=6
/proc/31739/stat : 31739 (march_hi) R 31734 31734 31124 0 -1 4202496 26085 0 0 0 59735 7 0 0 25 0 1 0 515002320 86335488 13672 33554432000 4194304 4331988 140733447867200 18446744073709551615 4228235 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/31739/statm: 21078 13672 175 34 0 19520 0
Current children cumulated CPU time (s) 4800.18
Current children cumulated vsize (KiB) 2771976

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 31734 and gives
#  childrusage.ru_utime.tv_sec=0
#  childrusage.ru_utime.tv_usec=1999
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=5999
# CPU time returned by wait4() is 0.007998
# while last known CPU time is 4800.18
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1201.71
CPU time (s): 4800.18
CPU user time (s): 4798.44
CPU system time (s): 1.74
CPU usage (%): 399.445
Max. virtual memory (cumulated for all children) (KiB): 4531744

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 0.001999
system time used= 0.005999
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 520
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 337
involuntary context switches= 6

runsolver used 1.90071 second user time and 4.66329 second system time

The end

Launcher Data 

Begin job on node104 at 2011-04-29 01:40:19
IDJOB=3258521
IDBENCH=71367
IDSOLVER=1612
FILE ID=node104/3258521-1304034019
RUNJOBID= node104-1304032539-31476
PBS_JOBID= 13170348
Free space on /tmp= 71532 MiB

SOLVER NAME= ppfolio par
BENCH NAME= SAT09/CRAFTED/rbsat/random/forced/rbsat-v2100c215164gyes7.cnf
COMMAND LINE= ppfolio BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3258521-1304034019/watcher-3258521-1304034019 -o /tmp/evaluation-result-3258521-1304034019/solver-3258521-1304034019 -C 4800 -W 1300 -M 15500  ppfolio HOME/instance-3258521-1304034019.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 140eea74f7e66501965365160d02fa4d
RANDOM SEED=1970536472

node104.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.71
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5293.18
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      16370920 kB
Buffers:       2207880 kB
Cached:       13436160 kB
SwapCached:          0 kB
Active:        3921676 kB
Inactive:     12069148 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      16370920 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:           12816 kB
Writeback:           0 kB
AnonPages:      345868 kB
Mapped:          16736 kB
Slab:           524140 kB
PageTables:       5712 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:  2617788 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 70488 MiB
End job on node104 at 2011-04-29 02:00:22