Trace number 3291468

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ppfolio par? (TO) 4800.25 1201.71

General information on the benchmark

NameSAT03/industrial/li/li-exam/
li-exam-61.shuffled-as.sat03-366.cnf
MD5SUM3be662940f394fa876137a905247d755
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables28147
Number of clauses108436
Sum of the clauses size373110
Maximum clause length166
Minimum clause length1
Number of clauses of size 114
Number of clauses of size 218786
Number of clauses of size 351588
Number of clauses of size 437440
Number of clauses of size 53
Number of clauses of size over 5605

Solver Data 

0.00/0.00	This is pico-portfolio, a naive parallel portfolio
0.00/0.00	Copyright (C) 2011 Olivier Roussel
0.00/0.00	Using 4 processing units
0.00/0.00	This portfolio uses the following solvers:
0.00/0.00	[1] cryptominisat 2.7.1 (Mate Soos)
0.00/0.00	[2] clasp 1.3.6 (Martin Gebser, Benjamin Kaufmann, and Torsten Schaub)
0.00/0.00	[3] TNM 2009 (Wanxia Wei and Chu Min Li)
0.00/0.00	[4] lingeling 276-6264d55-100731 (Armin Biere)
0.00/0.00	[5] march_hi 2009 (Marijn Heule and Hans Van Maaren)
0.00/0.00	Starting cryptominisat 2.7.1 on core(s) 0
0.00/0.00	Starting clasp 1.3.6 on core(s) 4
0.00/0.00	Starting TNM 2009 on core(s) 6
0.00/0.00	Starting lingeling 276-6264d55-100731 on core(s) 2
0.00/0.00	Starting march_hi 2009 on core(s) 6

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3291468-1304092831/watcher-3291468-1304092831 -o /tmp/evaluation-result-3291468-1304092831/solver-3291468-1304092831 -C 4800 -W 1300 -M 15500 ppfolio HOME/instance-3291468-1304092831.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 10.22 10.21 9.95 6/182 24702
/proc/meminfo: memFree=12549252/32951124 swapFree=67102140/67111528
[pid=24702] ppid=24700 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/24702/stat : 24702 (ppfolio) R 24700 24702 24128 0 -1 4202496 522 0 0 0 0 0 0 0 25 0 1 0 599513789 18763776 390 33554432000 4194304 4287625 140737372116880 18446744073709551615 216735198735 0 0 4096 65536 0 0 0 17 4 0 0 0
/proc/24702/statm: 4581 390 341 23 0 91 0

[startup+0.268664 s]
/proc/loadavg: 10.22 10.21 9.95 6/182 24702
/proc/meminfo: memFree=12549252/32951124 swapFree=67102140/67111528
[pid=24702] ppid=24700 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/24702/stat : 24702 (ppfolio) S 24700 24702 24128 0 -1 4202496 523 0 0 0 0 0 0 0 25 0 1 0 599513789 18763776 390 33554432000 4194304 4287625 140737372116880 18446744073709551615 216735198735 0 0 4096 65536 4598870252627951615 0 0 17 4 0 0 0
/proc/24702/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.300656 s]
/proc/loadavg: 10.22 10.21 9.95 6/182 24702
/proc/meminfo: memFree=12549252/32951124 swapFree=67102140/67111528
[pid=24702] ppid=24700 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/24702/stat : 24702 (ppfolio) S 24700 24702 24128 0 -1 4202496 523 0 0 0 0 0 0 0 25 0 1 0 599513789 18763776 390 33554432000 4194304 4287625 140737372116880 18446744073709551615 216735198735 0 0 4096 65536 4598870252627951615 0 0 17 4 0 0 0
/proc/24702/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.40064 s]
/proc/loadavg: 10.22 10.21 9.95 6/182 24702
/proc/meminfo: memFree=12549252/32951124 swapFree=67102140/67111528
[pid=24702] ppid=24700 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/24702/stat : 24702 (ppfolio) S 24700 24702 24128 0 -1 4202496 523 0 0 0 0 0 0 0 22 0 1 0 599513789 18763776 390 33554432000 4194304 4287625 140737372116880 18446744073709551615 216735198735 0 0 4096 65536 4598870252627951615 0 0 17 4 0 0 0
/proc/24702/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.700587 s]
/proc/loadavg: 10.22 10.21 9.95 6/182 24702
/proc/meminfo: memFree=12549252/32951124 swapFree=67102140/67111528
[pid=24702] ppid=24700 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/24702/stat : 24702 (ppfolio) S 24700 24702 24128 0 -1 4202496 523 0 0 0 0 0 0 0 22 0 1 0 599513789 18763776 390 33554432000 4194304 4287625 140737372116880 18446744073709551615 216735198735 0 0 4096 65536 4598870252627951615 0 0 17 4 0 0 0
/proc/24702/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+1.50045 s]
/proc/loadavg: 10.28 10.22 9.95 12/188 24708
/proc/meminfo: memFree=12451360/32951124 swapFree=67102140/67111528
[pid=24702] ppid=24700 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/24702/stat : 24702 (ppfolio) S 24700 24702 24128 0 -1 4202496 523 0 0 0 0 0 0 0 18 0 1 0 599513789 18763776 390 33554432000 4194304 4287625 140737372116880 18446744073709551615 216735198735 0 0 4096 65536 4598870252627951615 0 0 17 4 0 0 0
/proc/24702/statm: 4581 390 341 23 0 91 0
[pid=24703] ppid=24702 vsize=53840 CPUtime=1.48 cores=0
/proc/24703/stat : 24703 (cryptominisat) R 24702 24702 24128 0 -1 4202496 9952 0 0 0 147 1 0 0 25 0 1 0 599513789 55132160 8678 33554432000 4194304 4587354 140735014488176 18446744073709551615 4239534 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/24703/statm: 13460 8678 333 96 0 10022 0
[pid=24704] ppid=24702 vsize=33048 CPUtime=1.49 cores=4
/proc/24704/stat : 24704 (clasp) R 24702 24702 24128 0 -1 4202496 6877 0 0 0 149 0 0 0 25 0 1 0 599513789 33841152 5526 33554432000 4194304 4785740 140737132806272 18446744073709551615 4553003 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/24704/statm: 8262 5526 340 145 0 5307 0
[pid=24705] ppid=24702 vsize=464696 CPUtime=0.77 cores=6
/proc/24705/stat : 24705 (TNM) R 24702 24702 24128 0 -1 4202496 5130 0 0 0 77 0 0 0 25 0 1 0 599513789 475848704 5027 33554432000 134512640 134534564 4291259136 18446744073709551615 134529953 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/24705/statm: 116174 5027 95 6 0 114840 0
[pid=24706] ppid=24702 vsize=17392 CPUtime=1.48 cores=2
/proc/24706/stat : 24706 (lingeling) R 24702 24702 24128 0 -1 4202496 4738 0 0 0 148 0 0 0 25 0 1 0 599513789 17809408 3513 33554432000 4194304 4352236 140733238391856 18446744073709551615 4268336 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/24706/statm: 4348 3513 124 39 0 3428 0
[pid=24707] ppid=24702 vsize=25900 CPUtime=0.7 cores=6
/proc/24707/stat : 24707 (march_hi) R 24702 24702 24128 0 -1 4202496 16610 0 0 0 68 2 0 0 25 0 1 0 599513789 26521600 4064 33554432000 4194304 4331988 140735436800640 18446744073709551615 4217696 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/24707/statm: 6475 4064 157 34 0 4917 0
Current children cumulated CPU time (s) 5.92
Current children cumulated vsize (KiB) 613200

[startup+3.10117 s]
/proc/loadavg: 10.28 10.22 9.95 11/188 24708
/proc/meminfo: memFree=12440820/32951124 swapFree=67102140/67111528
[pid=24702] ppid=24700 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/24702/stat : 24702 (ppfolio) S 24700 24702 24128 0 -1 4202496 523 0 0 0 0 0 0 0 18 0 1 0 599513789 18763776 390 33554432000 4194304 4287625 140737372116880 18446744073709551615 216735198735 0 0 4096 65536 4598870252627951615 0 0 17 4 0 0 0
/proc/24702/statm: 4581 390 341 23 0 91 0
[pid=24703] ppid=24702 vsize=53840 CPUtime=3.08 cores=0
/proc/24703/stat : 24703 (cryptominisat) R 24702 24702 24128 0 -1 4202496 9985 0 0 0 307 1 0 0 25 0 1 0 599513789 55132160 8706 33554432000 4194304 4587354 140735014488176 18446744073709551615 4237679 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/24703/statm: 13460 8706 333 96 0 10022 0
[pid=24704] ppid=24702 vsize=33048 CPUtime=3.08 cores=4
/proc/24704/stat : 24704 (clasp) R 24702 24702 24128 0 -1 4202496 6877 0 0 0 308 0 0 0 25 0 1 0 599513789 33841152 5526 33554432000 4194304 4785740 140737132806272 18446744073709551615 4319146 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/24704/statm: 8262 5526 340 145 0 5307 0
[pid=24705] ppid=24702 vsize=464696 CPUtime=1.56 cores=6
/proc/24705/stat : 24705 (TNM) R 24702 24702 24128 0 -1 4202496 5130 0 0 0 156 0 0 0 25 0 1 0 599513789 475848704 5027 33554432000 134512640 134534564 4291259136 18446744073709551615 134529080 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/24705/statm: 116174 5027 95 6 0 114840 0
[pid=24706] ppid=24702 vsize=19100 CPUtime=3.08 cores=2
/proc/24706/stat : 24706 (lingeling) R 24702 24702 24128 0 -1 4202496 5165 0 0 0 308 0 0 0 25 0 1 0 599513789 19558400 3940 33554432000 4194304 4352236 140733238391856 18446744073709551615 4208229 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/24706/statm: 4775 3940 124 39 0 3855 0
[pid=24707] ppid=24702 vsize=29720 CPUtime=1.5 cores=6
/proc/24707/stat : 24707 (march_hi) R 24702 24702 24128 0 -1 4202496 18167 0 0 0 148 2 0 0 25 0 1 0 599513789 30433280 4932 33554432000 4194304 4331988 140735436800640 18446744073709551615 4224256 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/24707/statm: 7430 4932 157 34 0 5872 0
Current children cumulated CPU time (s) 12.3
Current children cumulated vsize (KiB) 618728

[startup+6.30061 s]
/proc/loadavg: 10.26 10.22 9.95 11/188 24708
/proc/meminfo: memFree=12419956/32951124 swapFree=67102140/67111528
[pid=24702] ppid=24700 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/24702/stat : 24702 (ppfolio) S 24700 24702 24128 0 -1 4202496 523 0 0 0 0 0 0 0 18 0 1 0 599513789 18763776 390 33554432000 4194304 4287625 140737372116880 18446744073709551615 216735198735 0 0 4096 65536 4598870252627951615 0 0 17 4 0 0 0
/proc/24702/statm: 4581 390 341 23 0 91 0
[pid=24703] ppid=24702 vsize=53840 CPUtime=6.28 cores=0
/proc/24703/stat : 24703 (cryptominisat) R 24702 24702 24128 0 -1 4202496 10033 0 0 0 627 1 0 0 25 0 1 0 599513789 55132160 8747 33554432000 4194304 4587354 140735014488176 18446744073709551615 4237665 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/24703/statm: 13460 8747 333 96 0 10022 0
[pid=24704] ppid=24702 vsize=33048 CPUtime=6.28 cores=4
/proc/24704/stat : 24704 (clasp) R 24702 24702 24128 0 -1 4202496 6877 0 0 0 628 0 0 0 25 0 1 0 599513789 33841152 5526 33554432000 4194304 4785740 140737132806272 18446744073709551615 4319149 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/24704/statm: 8262 5526 340 145 0 5307 0
[pid=24705] ppid=24702 vsize=464696 CPUtime=3.14 cores=6
/proc/24705/stat : 24705 (TNM) R 24702 24702 24128 0 -1 4202496 5130 0 0 0 314 0 0 0 25 0 1 0 599513789 475848704 5027 33554432000 134512640 134534564 4291259136 18446744073709551615 134528957 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/24705/statm: 116174 5027 95 6 0 114840 0
[pid=24706] ppid=24702 vsize=19100 CPUtime=6.28 cores=2
/proc/24706/stat : 24706 (lingeling) R 24702 24702 24128 0 -1 4202496 5167 0 0 0 628 0 0 0 25 0 1 0 599513789 19558400 3942 33554432000 4194304 4352236 140733238391856 18446744073709551615 4239164 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/24706/statm: 4775 3942 126 39 0 3855 0
[pid=24707] ppid=24702 vsize=67844 CPUtime=3.1 cores=6
/proc/24707/stat : 24707 (march_hi) R 24702 24702 24128 0 -1 4202496 27263 0 0 0 307 3 0 0 25 0 1 0 599513789 69472256 8652 33554432000 4194304 4331988 140735436800640 18446744073709551615 4228459 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/24707/statm: 16961 8652 184 34 0 15403 0
Current children cumulated CPU time (s) 25.08
Current children cumulated vsize (KiB) 656852

[startup+12.7005 s]
/proc/loadavg: 10.24 10.21 9.95 11/188 24708
/proc/meminfo: memFree=12407192/32951124 swapFree=67102140/67111528
[pid=24702] ppid=24700 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/24702/stat : 24702 (ppfolio) S 24700 24702 24128 0 -1 4202496 523 0 0 0 0 0 0 0 18 0 1 0 599513789 18763776 390 33554432000 4194304 4287625 140737372116880 18446744073709551615 216735198735 0 0 4096 65536 4598870252627951615 0 0 17 4 0 0 0
/proc/24702/statm: 4581 390 341 23 0 91 0
[pid=24703] ppid=24702 vsize=54108 CPUtime=12.69 cores=0
/proc/24703/stat : 24703 (cryptominisat) R 24702 24702 24128 0 -1 4202496 10645 0 0 0 1267 2 0 0 25 0 1 0 599513789 55406592 9354 33554432000 4194304 4587354 140735014488176 18446744073709551615 4238399 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/24703/statm: 13527 9354 333 96 0 10089 0
[pid=24704] ppid=24702 vsize=33048 CPUtime=12.68 cores=4
/proc/24704/stat : 24704 (clasp) R 24702 24702 24128 0 -1 4202496 6877 0 0 0 1268 0 0 0 25 0 1 0 599513789 33841152 5526 33554432000 4194304 4785740 140737132806272 18446744073709551615 4319012 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/24704/statm: 8262 5526 340 145 0 5307 0
[pid=24705] ppid=24702 vsize=464696 CPUtime=6.34 cores=6
/proc/24705/stat : 24705 (TNM) R 24702 24702 24128 0 -1 4202496 5130 0 0 0 634 0 0 0 25 0 1 0 599513789 475848704 5027 33554432000 134512640 134534564 4291259136 18446744073709551615 134528275 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/24705/statm: 116174 5027 95 6 0 114840 0
[pid=24706] ppid=24702 vsize=19100 CPUtime=12.68 cores=2
/proc/24706/stat : 24706 (lingeling) R 24702 24702 24128 0 -1 4202496 5167 0 0 0 1268 0 0 0 25 0 1 0 599513789 19558400 3942 33554432000 4194304 4352236 140733238391856 18446744073709551615 4328090 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/24706/statm: 4775 3942 126 39 0 3855 0
[pid=24707] ppid=24702 vsize=68260 CPUtime=6.27 cores=6
/proc/24707/stat : 24707 (march_hi) R 24702 24702 24128 0 -1 4202496 27374 0 0 0 624 3 0 0 25 0 1 0 599513789 69898240 8763 33554432000 4194304 4331988 140735436800640 18446744073709551615 4228459 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/24707/statm: 17065 8763 184 34 0 15507 0
Current children cumulated CPU time (s) 50.66
Current children cumulated vsize (KiB) 657536

[startup+25.5063 s]
/proc/loadavg: 10.18 10.20 9.95 11/188 24708
/proc/meminfo: memFree=12409308/32951124 swapFree=67102140/67111528
[pid=24702] ppid=24700 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/24702/stat : 24702 (ppfolio) S 24700 24702 24128 0 -1 4202496 523 0 0 0 0 0 0 0 15 0 1 0 599513789 18763776 390 33554432000 4194304 4287625 140737372116880 18446744073709551615 216735198735 0 0 4096 65536 4598870252627951615 0 0 17 4 0 0 0
/proc/24702/statm: 4581 390 341 23 0 91 0
[pid=24703] ppid=24702 vsize=188956 CPUtime=25.5 cores=0
/proc/24703/stat : 24703 (cryptominisat) R 24702 24702 24128 0 -1 4202496 19667 0 0 0 2545 5 0 0 25 0 1 0 599513789 193490944 11756 33554432000 4194304 4587354 140735014488176 18446744073709551615 4237679 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/24703/statm: 47239 11756 339 96 0 43801 0
[pid=24704] ppid=24702 vsize=39848 CPUtime=25.49 cores=4
/proc/24704/stat : 24704 (clasp) R 24702 24702 24128 0 -1 4202496 8608 0 0 0 2548 1 0 0 25 0 1 0 599513789 40804352 7257 33554432000 4194304 4785740 140737132806272 18446744073709551615 4287028 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/24704/statm: 9962 7257 340 145 0 7007 0
[pid=24705] ppid=24702 vsize=464696 CPUtime=12.74 cores=6
/proc/24705/stat : 24705 (TNM) R 24702 24702 24128 0 -1 4202496 5130 0 0 0 1274 0 0 0 25 0 1 0 599513789 475848704 5027 33554432000 134512640 134534564 4291259136 18446744073709551615 134525873 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/24705/statm: 116174 5027 95 6 0 114840 0
[pid=24706] ppid=24702 vsize=18472 CPUtime=25.49 cores=2
/proc/24706/stat : 24706 (lingeling) R 24702 24702 24128 0 -1 4202496 5639 0 0 0 2549 0 0 0 25 0 1 0 599513789 18915328 3785 33554432000 4194304 4352236 140733238391856 18446744073709551615 4327820 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/24706/statm: 4618 3785 126 39 0 3698 0
[pid=24707] ppid=24702 vsize=68488 CPUtime=12.6 cores=6
/proc/24707/stat : 24707 (march_hi) R 24702 24702 24128 0 -1 4202496 27400 0 0 0 1257 3 0 0 25 0 1 0 599513789 70131712 8789 33554432000 4194304 4331988 140735436800640 18446744073709551615 4228459 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/24707/statm: 17122 8789 184 34 0 15564 0
Current children cumulated CPU time (s) 101.82
Current children cumulated vsize (KiB) 798784

[startup+51.1008 s]
/proc/loadavg: 10.31 10.23 9.97 11/188 24718
/proc/meminfo: memFree=12346132/32951124 swapFree=67102140/67111528
[pid=24702] ppid=24700 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/24702/stat : 24702 (ppfolio) S 24700 24702 24128 0 -1 4202496 523 0 0 0 0 0 0 0 15 0 1 0 599513789 18763776 390 33554432000 4194304 4287625 140737372116880 18446744073709551615 216735198735 0 0 4096 65536 4598870252627951615 0 0 17 4 0 0 0
/proc/24702/statm: 4581 390 341 23 0 91 0
[pid=24703] ppid=24702 vsize=222380 CPUtime=51.09 cores=0
/proc/24703/stat : 24703 (cryptominisat) R 24702 24702 24128 0 -1 4202496 50658 0 0 0 5096 13 0 0 25 0 1 0 599513789 227717120 13246 33554432000 4194304 4587354 140735014488176 18446744073709551615 4237928 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/24703/statm: 55595 13246 339 96 0 52157 0
[pid=24704] ppid=24702 vsize=78440 CPUtime=51.09 cores=4
/proc/24704/stat : 24704 (clasp) R 24702 24702 24128 0 -1 4202496 18219 0 0 0 5105 4 0 0 25 0 1 0 599513789 80322560 16868 33554432000 4194304 4785740 140737132806272 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/24704/statm: 19610 16868 340 145 0 16655 0
[pid=24705] ppid=24702 vsize=464696 CPUtime=25.44 cores=6
/proc/24705/stat : 24705 (TNM) R 24702 24702 24128 0 -1 4202496 5130 0 0 0 2544 0 0 0 25 0 1 0 599513789 475848704 5027 33554432000 134512640 134534564 4291259136 18446744073709551615 134530438 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/24705/statm: 116174 5027 95 6 0 114840 0
[pid=24706] ppid=24702 vsize=22396 CPUtime=51.08 cores=2

################
# More data... #
################

/proc/24707/statm: 18720 10280 184 34 0 17162 0
Current children cumulated CPU time (s) 3764.02
Current children cumulated vsize (KiB) 1297772

[startup+1002.31 s]
/proc/loadavg: 10.18 10.17 10.09 11/188 24873
/proc/meminfo: memFree=11782244/32951124 swapFree=67102140/67111528
[pid=24702] ppid=24700 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/24702/stat : 24702 (ppfolio) S 24700 24702 24128 0 -1 4202496 523 0 0 0 0 0 0 0 15 0 1 0 599513789 18763776 390 33554432000 4194304 4287625 140737372116880 18446744073709551615 216735198735 0 0 4096 65536 4598870252627951615 0 0 17 4 0 0 0
/proc/24702/statm: 4581 390 341 23 0 91 0
[pid=24703] ppid=24702 vsize=679048 CPUtime=1002.31 cores=0
/proc/24703/stat : 24703 (cryptominisat) R 24702 24702 24128 0 -1 4202496 672610 0 0 0 99998 233 0 0 25 0 1 0 599513789 695345152 43047 33554432000 4194304 4587354 140735014488176 18446744073709551615 4236491 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/24703/statm: 169762 43047 339 96 0 166324 0
[pid=24704] ppid=24702 vsize=342944 CPUtime=1002.3 cores=4
/proc/24704/stat : 24704 (clasp) R 24702 24702 24128 0 -1 4202496 83707 0 0 0 100208 22 0 0 25 0 1 0 599513789 351174656 82356 33554432000 4194304 4785740 140737132806272 18446744073709551615 4287204 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/24704/statm: 85736 82356 340 145 0 82781 0
[pid=24705] ppid=24702 vsize=464696 CPUtime=498.46 cores=6
/proc/24705/stat : 24705 (TNM) R 24702 24702 24128 0 -1 4202496 5130 0 0 0 49846 0 0 0 25 0 1 0 599513789 475848704 5027 33554432000 134512640 134534564 4291259136 18446744073709551615 6188052 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/24705/statm: 116174 5027 95 6 0 114840 0
[pid=24706] ppid=24702 vsize=43364 CPUtime=1002.28 cores=2
/proc/24706/stat : 24706 (lingeling) R 24702 24702 24128 0 -1 4202496 37049 0 0 0 100223 5 0 0 25 0 1 0 599513789 44404736 10009 33554432000 4194304 4352236 140733238391856 18446744073709551615 4328035 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/24706/statm: 10841 10009 127 39 0 9921 0
[pid=24707] ppid=24702 vsize=74880 CPUtime=498.37 cores=6
/proc/24707/stat : 24707 (march_hi) R 24702 24702 24128 0 -1 4202496 28967 0 0 0 49833 4 0 0 25 0 1 0 599513789 76677120 10356 33554432000 4194304 4331988 140735436800640 18446744073709551615 4265456 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/24707/statm: 18720 10356 184 34 0 17162 0
Current children cumulated CPU time (s) 4003.72
Current children cumulated vsize (KiB) 1623256

[startup+1062.3 s]
/proc/loadavg: 10.18 10.16 10.09 11/188 24874
/proc/meminfo: memFree=11752344/32951124 swapFree=67102140/67111528
[pid=24702] ppid=24700 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/24702/stat : 24702 (ppfolio) S 24700 24702 24128 0 -1 4202496 523 0 0 0 0 0 0 0 15 0 1 0 599513789 18763776 390 33554432000 4194304 4287625 140737372116880 18446744073709551615 216735198735 0 0 4096 65536 4598870252627951615 0 0 17 4 0 0 0
/proc/24702/statm: 4581 390 341 23 0 91 0
[pid=24703] ppid=24702 vsize=282992 CPUtime=1062.31 cores=0
/proc/24703/stat : 24703 (cryptominisat) R 24702 24702 24128 0 -1 4202496 702042 0 0 0 105986 245 0 0 25 0 1 0 599513789 289783808 47809 33554432000 4194304 4587354 140735014488176 18446744073709551615 4238690 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/24703/statm: 70748 47809 339 96 0 67310 0
[pid=24704] ppid=24702 vsize=354176 CPUtime=1062.29 cores=4
/proc/24704/stat : 24704 (clasp) R 24702 24702 24128 0 -1 4202496 86557 0 0 0 106207 22 0 0 25 0 1 0 599513789 362676224 85206 33554432000 4194304 4785740 140737132806272 18446744073709551615 4552550 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/24704/statm: 88544 85206 340 145 0 85589 0
[pid=24705] ppid=24702 vsize=464696 CPUtime=528.28 cores=6
/proc/24705/stat : 24705 (TNM) R 24702 24702 24128 0 -1 4202496 5130 0 0 0 52828 0 0 0 25 0 1 0 599513789 475848704 5027 33554432000 134512640 134534564 4291259136 18446744073709551615 134528957 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/24705/statm: 116174 5027 95 6 0 114840 0
[pid=24706] ppid=24702 vsize=39268 CPUtime=1062.27 cores=2
/proc/24706/stat : 24706 (lingeling) R 24702 24702 24128 0 -1 4202496 39097 0 0 0 106222 5 0 0 25 0 1 0 599513789 40210432 8985 33554432000 4194304 4352236 140733238391856 18446744073709551615 4328991 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/24706/statm: 9817 8985 127 39 0 8897 0
[pid=24707] ppid=24702 vsize=77468 CPUtime=528.22 cores=6
/proc/24707/stat : 24707 (march_hi) R 24702 24702 24128 0 -1 4202496 29040 0 0 0 52818 4 0 0 25 0 1 0 599513789 79327232 10429 33554432000 4194304 4331988 140735436800640 18446744073709551615 4210940 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/24707/statm: 19367 10429 184 34 0 17809 0
Current children cumulated CPU time (s) 4243.37
Current children cumulated vsize (KiB) 1236924

[startup+1122.3 s]
/proc/loadavg: 10.20 10.18 10.10 11/188 24876
/proc/meminfo: memFree=11703096/32951124 swapFree=67102140/67111528
[pid=24702] ppid=24700 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/24702/stat : 24702 (ppfolio) S 24700 24702 24128 0 -1 4202496 523 0 0 0 0 0 0 0 15 0 1 0 599513789 18763776 390 33554432000 4194304 4287625 140737372116880 18446744073709551615 216735198735 0 0 4096 65536 4598870252627951615 0 0 17 4 0 0 0
/proc/24702/statm: 4581 390 341 23 0 91 0
[pid=24703] ppid=24702 vsize=287608 CPUtime=1122.31 cores=0
/proc/24703/stat : 24703 (cryptominisat) R 24702 24702 24128 0 -1 4202496 738327 0 0 0 111971 260 0 0 25 0 1 0 599513789 294510592 42589 33554432000 4194304 4587354 140735014488176 18446744073709551615 4238463 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/24703/statm: 71902 42589 339 96 0 68464 0
[pid=24704] ppid=24702 vsize=400400 CPUtime=1122.3 cores=4
/proc/24704/stat : 24704 (clasp) R 24702 24702 24128 0 -1 4202496 97999 0 0 0 112205 25 0 0 25 0 1 0 599513789 410009600 96648 33554432000 4194304 4785740 140737132806272 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/24704/statm: 100100 96648 340 145 0 97145 0
[pid=24705] ppid=24702 vsize=464696 CPUtime=558.15 cores=6
/proc/24705/stat : 24705 (TNM) R 24702 24702 24128 0 -1 4202496 5130 0 0 0 55815 0 0 0 25 0 1 0 599513789 475848704 5027 33554432000 134512640 134534564 4291259136 18446744073709551615 134528726 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/24705/statm: 116174 5027 95 6 0 114840 0
[pid=24706] ppid=24702 vsize=51556 CPUtime=1122.28 cores=2
/proc/24706/stat : 24706 (lingeling) R 24702 24702 24128 0 -1 4202496 43193 0 0 0 112222 6 0 0 25 0 1 0 599513789 52793344 12057 33554432000 4194304 4352236 140733238391856 18446744073709551615 4329086 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/24706/statm: 12889 12057 127 39 0 11969 0
[pid=24707] ppid=24702 vsize=77468 CPUtime=558.02 cores=6
/proc/24707/stat : 24707 (march_hi) R 24702 24702 24128 0 -1 4202496 29120 0 0 0 55798 4 0 0 25 0 1 0 599513789 79327232 10509 33554432000 4194304 4331988 140735436800640 18446744073709551615 4228459 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/24707/statm: 19367 10509 184 34 0 17809 0
Current children cumulated CPU time (s) 4483.06
Current children cumulated vsize (KiB) 1300052

[startup+1182.3 s]
/proc/loadavg: 10.26 10.19 10.10 11/188 24877
/proc/meminfo: memFree=11652364/32951124 swapFree=67102140/67111528
[pid=24702] ppid=24700 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/24702/stat : 24702 (ppfolio) S 24700 24702 24128 0 -1 4202496 523 0 0 0 0 0 0 0 15 0 1 0 599513789 18763776 390 33554432000 4194304 4287625 140737372116880 18446744073709551615 216735198735 0 0 4096 65536 4598870252627951615 0 0 17 4 0 0 0
/proc/24702/statm: 4581 390 341 23 0 91 0
[pid=24703] ppid=24702 vsize=325824 CPUtime=1182.31 cores=0
/proc/24703/stat : 24703 (cryptominisat) R 24702 24702 24128 0 -1 4202496 778075 0 0 0 117956 275 0 0 25 0 1 0 599513789 333643776 53280 33554432000 4194304 4587354 140735014488176 18446744073709551615 4238611 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/24703/statm: 81456 53280 339 96 0 78018 0
[pid=24704] ppid=24702 vsize=400400 CPUtime=1182.3 cores=4
/proc/24704/stat : 24704 (clasp) R 24702 24702 24128 0 -1 4202496 98011 0 0 0 118205 25 0 0 25 0 1 0 599513789 410009600 96660 33554432000 4194304 4785740 140737132806272 18446744073709551615 4286941 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/24704/statm: 100100 96660 340 145 0 97145 0
[pid=24705] ppid=24702 vsize=464696 CPUtime=587.95 cores=6
/proc/24705/stat : 24705 (TNM) R 24702 24702 24128 0 -1 4202496 5130 0 0 0 58795 0 0 0 25 0 1 0 599513789 475848704 5027 33554432000 134512640 134534564 4291259136 18446744073709551615 134530632 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/24705/statm: 116174 5027 95 6 0 114840 0
[pid=24706] ppid=24702 vsize=43364 CPUtime=1182.28 cores=2
/proc/24706/stat : 24706 (lingeling) R 24702 24702 24128 0 -1 4202496 43705 0 0 0 118222 6 0 0 25 0 1 0 599513789 44404736 10009 33554432000 4194304 4352236 140733238391856 18446744073709551615 4328194 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/24706/statm: 10841 10009 127 39 0 9921 0
[pid=24707] ppid=24702 vsize=77468 CPUtime=587.9 cores=6
/proc/24707/stat : 24707 (march_hi) R 24702 24702 24128 0 -1 4202496 29190 0 0 0 58786 4 0 0 25 0 1 0 599513789 79327232 10579 33554432000 4194304 4331988 140735436800640 18446744073709551615 4226434 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/24707/statm: 19367 10579 184 34 0 17809 0
Current children cumulated CPU time (s) 4722.74
Current children cumulated vsize (KiB) 1330076



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1201.7 s]
/proc/loadavg: 10.19 10.18 10.10 11/188 24878
/proc/meminfo: memFree=11619000/32951124 swapFree=67102140/67111528
[pid=24702] ppid=24700 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/24702/stat : 24702 (ppfolio) S 24700 24702 24128 0 -1 4202496 523 0 0 0 0 0 0 0 15 0 1 0 599513789 18763776 390 33554432000 4194304 4287625 140737372116880 18446744073709551615 216735198735 0 0 4096 65536 4598870252627951615 0 0 17 4 0 0 0
/proc/24702/statm: 4581 390 341 23 0 91 0
[pid=24703] ppid=24702 vsize=325824 CPUtime=1201.72 cores=0
/proc/24703/stat : 24703 (cryptominisat) R 24702 24702 24128 0 -1 4202496 789064 0 0 0 119893 279 0 0 25 0 1 0 599513789 333643776 64248 33554432000 4194304 4587354 140735014488176 18446744073709551615 4238568 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/24703/statm: 81456 64248 339 96 0 78018 0
[pid=24704] ppid=24702 vsize=400400 CPUtime=1201.71 cores=4
/proc/24704/stat : 24704 (clasp) R 24702 24702 24128 0 -1 4202496 98021 0 0 0 120146 25 0 0 25 0 1 0 599513789 410009600 96670 33554432000 4194304 4785740 140737132806272 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/24704/statm: 100100 96670 340 145 0 97145 0
[pid=24705] ppid=24702 vsize=464696 CPUtime=597.63 cores=6
/proc/24705/stat : 24705 (TNM) R 24702 24702 24128 0 -1 4202496 5130 0 0 0 59763 0 0 0 25 0 1 0 599513789 475848704 5027 33554432000 134512640 134534564 4291259136 18446744073709551615 134525904 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/24705/statm: 116174 5027 95 6 0 114840 0
[pid=24706] ppid=24702 vsize=45292 CPUtime=1201.68 cores=2
/proc/24706/stat : 24706 (lingeling) R 24702 24702 24128 0 -1 4202496 44187 0 0 0 120162 6 0 0 25 0 1 0 599513789 46379008 10491 33554432000 4194304 4352236 140733238391856 18446744073709551615 4329328 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/24706/statm: 11323 10491 127 39 0 10403 0
[pid=24707] ppid=24702 vsize=77468 CPUtime=597.51 cores=6
/proc/24707/stat : 24707 (march_hi) R 24702 24702 24128 0 -1 4202496 29214 0 0 0 59747 4 0 0 25 0 1 0 599513789 79327232 10603 33554432000 4194304 4331988 140735436800640 18446744073709551615 4211208 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/24707/statm: 19367 10603 184 34 0 17809 0
Current children cumulated CPU time (s) 4800.25
Current children cumulated vsize (KiB) 1332004

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1201.7 s]
/proc/loadavg: 10.19 10.18 10.10 11/188 24878
/proc/meminfo: memFree=11619000/32951124 swapFree=67102140/67111528
[pid=24702] ppid=24700 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/24702/stat : 24702 (ppfolio) S 24700 24702 24128 0 -1 4202496 523 0 0 0 0 0 0 0 15 0 1 0 599513789 18763776 390 33554432000 4194304 4287625 140737372116880 18446744073709551615 216735198735 0 0 4096 65536 4598870252627951615 0 0 17 4 0 0 0
/proc/24702/statm: 4581 390 341 23 0 91 0
[pid=24703] ppid=24702 vsize=325824 CPUtime=1201.72 cores=0
/proc/24703/stat : 24703 (cryptominisat) R 24702 24702 24128 0 -1 4202496 789064 0 0 0 119893 279 0 0 25 0 1 0 599513789 333643776 64248 33554432000 4194304 4587354 140735014488176 18446744073709551615 4238568 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/24703/statm: 81456 64248 339 96 0 78018 0
[pid=24704] ppid=24702 vsize=400400 CPUtime=1201.71 cores=4
/proc/24704/stat : 24704 (clasp) R 24702 24702 24128 0 -1 4202496 98021 0 0 0 120146 25 0 0 25 0 1 0 599513789 410009600 96670 33554432000 4194304 4785740 140737132806272 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/24704/statm: 100100 96670 340 145 0 97145 0
[pid=24705] ppid=24702 vsize=464696 CPUtime=597.63 cores=6
/proc/24705/stat : 24705 (TNM) R 24702 24702 24128 0 -1 4202496 5130 0 0 0 59763 0 0 0 25 0 1 0 599513789 475848704 5027 33554432000 134512640 134534564 4291259136 18446744073709551615 134525904 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/24705/statm: 116174 5027 95 6 0 114840 0
[pid=24706] ppid=24702 vsize=45292 CPUtime=1201.68 cores=2
/proc/24706/stat : 24706 (lingeling) R 24702 24702 24128 0 -1 4202496 44187 0 0 0 120162 6 0 0 25 0 1 0 599513789 46379008 10491 33554432000 4194304 4352236 140733238391856 18446744073709551615 4329328 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/24706/statm: 11323 10491 127 39 0 10403 0
[pid=24707] ppid=24702 vsize=77468 CPUtime=597.51 cores=6
/proc/24707/stat : 24707 (march_hi) R 24702 24702 24128 0 -1 4202496 29214 0 0 0 59747 4 0 0 25 0 1 0 599513789 79327232 10603 33554432000 4194304 4331988 140735436800640 18446744073709551615 4211208 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/24707/statm: 19367 10603 184 34 0 17809 0
Current children cumulated CPU time (s) 4800.25
Current children cumulated vsize (KiB) 1332004

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 24702 and gives
#  childrusage.ru_utime.tv_sec=0
#  childrusage.ru_utime.tv_usec=1999
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=0
# CPU time returned by wait4() is 0.001999
# while last known CPU time is 4800.25
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1201.71
CPU time (s): 4800.25
CPU user time (s): 4797.11
CPU system time (s): 3.14
CPU usage (%): 399.451
Max. virtual memory (cumulated for all children) (KiB): 1827428

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 0.001999
system time used= 0
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 523
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 173
involuntary context switches= 6

runsolver used 1.89171 second user time and 4.67329 second system time

The end

Launcher Data 

Begin job on node119 at 2011-04-29 18:00:31
IDJOB=3291468
IDBENCH=84223
IDSOLVER=1612
FILE ID=node119/3291468-1304092831
RUNJOBID= node119-1304090423-24477
PBS_JOBID= 13170550
Free space on /tmp= 71804 MiB

SOLVER NAME= ppfolio par
BENCH NAME= SAT03/industrial/li/li-exam/li-exam-61.shuffled-as.sat03-366.cnf
COMMAND LINE= ppfolio BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3291468-1304092831/watcher-3291468-1304092831 -o /tmp/evaluation-result-3291468-1304092831/solver-3291468-1304092831 -C 4800 -W 1300 -M 15500  ppfolio HOME/instance-3291468-1304092831.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 3be662940f394fa876137a905247d755
RANDOM SEED=1090438039

node119.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.63
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      12549416 kB
Buffers:       2235020 kB
Cached:       17325876 kB
SwapCached:       5504 kB
Active:        8184688 kB
Inactive:     11631196 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      12549416 kB
SwapTotal:    67111528 kB
SwapFree:     67102140 kB
Dirty:           12456 kB
Writeback:           0 kB
AnonPages:      252424 kB
Mapped:          16564 kB
Slab:           521712 kB
PageTables:       5568 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:  2481892 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71792 MiB
End job on node119 at 2011-04-29 18:20:35