Trace number 3280290

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ppfolio par? (TO) 4800.18 1201.71

General information on the benchmark

Nameindustrial/anbulagan/
hard-unsat/total-10-17-u.cnf
MD5SUM9ce7c48a28ddd5790e38607bb893dd38
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables293864
Number of clauses1370917
Sum of the clauses size3163902
Maximum clause length10
Minimum clause length1
Number of clauses of size 124516
Number of clauses of size 21065117
Number of clauses of size 3226184
Number of clauses of size 46120
Number of clauses of size 524480
Number of clauses of size over 524500

Solver Data 

0.00/0.00	This is pico-portfolio, a naive parallel portfolio
0.00/0.00	Copyright (C) 2011 Olivier Roussel
0.00/0.00	Using 4 processing units
0.00/0.00	This portfolio uses the following solvers:
0.00/0.00	[1] cryptominisat 2.7.1 (Mate Soos)
0.00/0.00	[2] clasp 1.3.6 (Martin Gebser, Benjamin Kaufmann, and Torsten Schaub)
0.00/0.00	[3] TNM 2009 (Wanxia Wei and Chu Min Li)
0.00/0.00	[4] lingeling 276-6264d55-100731 (Armin Biere)
0.00/0.00	[5] march_hi 2009 (Marijn Heule and Hans Van Maaren)
0.00/0.00	Starting cryptominisat 2.7.1 on core(s) 0
0.00/0.00	Starting clasp 1.3.6 on core(s) 4
0.00/0.00	Starting TNM 2009 on core(s) 6
0.00/0.00	Starting lingeling 276-6264d55-100731 on core(s) 2
0.00/0.00	Starting march_hi 2009 on core(s) 6

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3280290-1304029894/watcher-3280290-1304029894 -o /tmp/evaluation-result-3280290-1304029894/solver-3280290-1304029894 -C 4800 -W 1300 -M 15500 ppfolio HOME/instance-3280290-1304029894.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 9.76 9.92 7.34 6/182 7408
/proc/meminfo: memFree=14150096/32951124 swapFree=67111528/67111528
[pid=7408] ppid=7406 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/7408/stat : 7408 (ppfolio) R 7406 7408 7235 0 -1 4202496 522 0 0 0 0 0 0 0 25 0 1 0 514589950 18763776 390 33554432000 4194304 4287625 140736069988608 18446744073709551615 244446965263 0 0 4096 65536 0 0 0 17 4 0 0 0
/proc/7408/statm: 4581 390 341 23 0 91 0

[startup+0.0222839 s]
/proc/loadavg: 9.76 9.92 7.34 6/182 7408
/proc/meminfo: memFree=14150096/32951124 swapFree=67111528/67111528
[pid=7408] ppid=7406 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/7408/stat : 7408 (ppfolio) R 7406 7408 7235 0 -1 4202496 522 0 0 0 0 0 0 0 25 0 1 0 514589950 18763776 390 33554432000 4194304 4287625 140736069988608 18446744073709551615 244446965263 0 0 4096 65536 0 0 0 17 4 0 0 0
/proc/7408/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.10027 s]
/proc/loadavg: 9.76 9.92 7.34 6/182 7408
/proc/meminfo: memFree=14150096/32951124 swapFree=67111528/67111528
[pid=7408] ppid=7406 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/7408/stat : 7408 (ppfolio) R 7406 7408 7235 0 -1 4202496 522 0 0 0 0 0 0 0 25 0 1 0 514589950 18763776 390 33554432000 4194304 4287625 140736069988608 18446744073709551615 244446965263 0 0 4096 65536 0 0 0 17 4 0 0 0
/proc/7408/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.300233 s]
/proc/loadavg: 9.76 9.92 7.34 6/182 7408
/proc/meminfo: memFree=14150096/32951124 swapFree=67111528/67111528
[pid=7408] ppid=7406 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/7408/stat : 7408 (ppfolio) S 7406 7408 7235 0 -1 4202496 522 0 0 0 0 0 0 0 25 0 1 0 514589950 18763776 390 33554432000 4194304 4287625 140736069988608 18446744073709551615 244446965263 0 0 4096 65536 6965881017840173055 0 0 17 4 0 0 0
/proc/7408/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.70016 s]
/proc/loadavg: 9.76 9.92 7.34 6/182 7408
/proc/meminfo: memFree=14150096/32951124 swapFree=67111528/67111528
[pid=7408] ppid=7406 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/7408/stat : 7408 (ppfolio) S 7406 7408 7235 0 -1 4202496 522 0 0 0 0 0 0 0 21 0 1 0 514589950 18763776 390 33554432000 4194304 4287625 140736069988608 18446744073709551615 244446965263 0 0 4096 65536 6965881017840173055 0 0 17 4 0 0 0
/proc/7408/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+1.50101 s]
/proc/loadavg: 9.76 9.92 7.34 11/188 7414
/proc/meminfo: memFree=13728944/32951124 swapFree=67111528/67111528
[pid=7408] ppid=7406 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/7408/stat : 7408 (ppfolio) S 7406 7408 7235 0 -1 4202496 522 0 0 0 0 0 0 0 21 0 1 0 514589950 18763776 390 33554432000 4194304 4287625 140736069988608 18446744073709551615 244446965263 0 0 4096 65536 6965881017840173055 0 0 17 4 0 0 0
/proc/7408/statm: 4581 390 341 23 0 91 0
[pid=7409] ppid=7408 vsize=158000 CPUtime=1.49 cores=0
/proc/7409/stat : 7409 (cryptominisat) R 7408 7408 7235 0 -1 4202496 34486 0 0 0 144 5 0 0 20 0 1 0 514589950 161792000 33584 33554432000 4194304 4587354 140733601901696 18446744073709551615 4237467 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/7409/statm: 39500 33584 306 96 0 36062 0
[pid=7410] ppid=7408 vsize=155712 CPUtime=1.49 cores=4
/proc/7410/stat : 7410 (clasp) R 7408 7408 7235 0 -1 4202496 44476 0 0 0 144 5 0 0 25 0 1 0 514589950 159449088 34943 33554432000 4194304 4785740 140734651353680 18446744073709551615 4494020 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/7410/statm: 38928 34943 319 145 0 35973 0
[pid=7411] ppid=7408 vsize=470876 CPUtime=0.74 cores=6
/proc/7411/stat : 7411 (TNM) R 7408 7408 7235 0 -1 4202496 8291 0 0 0 72 2 0 0 25 0 1 0 514589950 482177024 8189 33554432000 134512640 134534564 4292633392 18446744073709551615 9898942 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/7411/statm: 117719 8189 90 6 0 116385 0
[pid=7412] ppid=7408 vsize=68108 CPUtime=1.48 cores=2
/proc/7412/stat : 7412 (lingeling) R 7408 7408 7235 0 -1 4202496 20362 0 0 0 145 3 0 0 25 0 1 0 514589950 69742592 16204 33554432000 4194304 4352236 140737120871168 18446744073709551615 4234513 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/7412/statm: 17027 16204 110 39 0 16107 0
[pid=7413] ppid=7408 vsize=94756 CPUtime=0.73 cores=6
/proc/7413/stat : 7413 (march_hi) R 7408 7408 7235 0 -1 4194304 21999 0 0 0 70 3 0 0 25 0 1 0 514589950 97030144 21894 33554432000 4194304 4331988 140734996688624 18446744073709551615 4260962 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/7413/statm: 23689 21894 113 34 0 22131 0
Current children cumulated CPU time (s) 5.93
Current children cumulated vsize (KiB) 965776

[startup+3.10072 s]
/proc/loadavg: 9.76 9.92 7.34 11/188 7414
/proc/meminfo: memFree=13549516/32951124 swapFree=67111528/67111528
[pid=7408] ppid=7406 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/7408/stat : 7408 (ppfolio) S 7406 7408 7235 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 514589950 18763776 390 33554432000 4194304 4287625 140736069988608 18446744073709551615 244446965263 0 0 4096 65536 6965881017840173055 0 0 17 4 0 0 0
/proc/7408/statm: 4581 390 341 23 0 91 0
[pid=7409] ppid=7408 vsize=210364 CPUtime=3.09 cores=0
/proc/7409/stat : 7409 (cryptominisat) R 7408 7408 7235 0 -1 4202496 47486 0 0 0 301 8 0 0 25 0 1 0 514589950 215412736 46581 33554432000 4194304 4587354 140733601901696 18446744073709551615 4421160 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/7409/statm: 52591 46581 312 96 0 49153 0
[pid=7410] ppid=7408 vsize=131928 CPUtime=3.09 cores=4
/proc/7410/stat : 7410 (clasp) R 7408 7408 7235 0 -1 4202496 44547 0 0 0 304 5 0 0 25 0 1 0 514589950 135094272 29514 33554432000 4194304 4785740 140734651353680 18446744073709551615 4553003 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/7410/statm: 32982 29514 333 145 0 30027 0
[pid=7411] ppid=7408 vsize=529084 CPUtime=1.53 cores=6
/proc/7411/stat : 7411 (TNM) R 7408 7408 7235 0 -1 4202496 26356 0 0 0 149 4 0 0 25 0 1 0 514589950 541782016 26253 33554432000 134512640 134534564 4292633392 18446744073709551615 134524554 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/7411/statm: 132271 26253 91 6 0 130937 0
[pid=7412] ppid=7408 vsize=67792 CPUtime=3.08 cores=2
/proc/7412/stat : 7412 (lingeling) R 7408 7408 7235 0 -1 4202496 31145 0 0 0 302 6 0 0 25 0 1 0 514589950 69419008 16102 33554432000 4194304 4352236 140737120871168 18446744073709551615 4319056 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/7412/statm: 16948 16102 119 39 0 16028 0
[pid=7413] ppid=7408 vsize=229904 CPUtime=1.53 cores=6
/proc/7413/stat : 7413 (march_hi) R 7408 7408 7235 0 -1 4194304 54953 0 0 0 145 8 0 0 25 0 1 0 514589950 235421696 42986 33554432000 4194304 4331988 140734996688624 18446744073709551615 4252157 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/7413/statm: 57476 42986 197 34 0 55918 0
Current children cumulated CPU time (s) 12.32
Current children cumulated vsize (KiB) 1187396

[startup+6.30112 s]
/proc/loadavg: 9.78 9.92 7.36 11/188 7414
/proc/meminfo: memFree=13436188/32951124 swapFree=67111528/67111528
[pid=7408] ppid=7406 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/7408/stat : 7408 (ppfolio) S 7406 7408 7235 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 514589950 18763776 390 33554432000 4194304 4287625 140736069988608 18446744073709551615 244446965263 0 0 4096 65536 6965881017840173055 0 0 17 4 0 0 0
/proc/7408/statm: 4581 390 341 23 0 91 0
[pid=7409] ppid=7408 vsize=211416 CPUtime=6.29 cores=0
/proc/7409/stat : 7409 (cryptominisat) R 7408 7408 7235 0 -1 4202496 47718 0 0 0 621 8 0 0 25 0 1 0 514589950 216489984 46813 33554432000 4194304 4587354 140733601901696 18446744073709551615 4421069 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/7409/statm: 52854 46813 312 96 0 49416 0
[pid=7410] ppid=7408 vsize=131928 CPUtime=6.29 cores=4
/proc/7410/stat : 7410 (clasp) R 7408 7408 7235 0 -1 4202496 44573 0 0 0 624 5 0 0 25 0 1 0 514589950 135094272 29540 33554432000 4194304 4785740 140734651353680 18446744073709551615 4553003 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/7410/statm: 32982 29540 340 145 0 30027 0
[pid=7411] ppid=7408 vsize=542284 CPUtime=3.14 cores=6
/proc/7411/stat : 7411 (TNM) R 7408 7408 7235 0 -1 4202496 41233 0 0 0 307 7 0 0 25 0 1 0 514589950 555298816 41130 33554432000 134512640 134534564 4292633392 18446744073709551615 134529352 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/7411/statm: 135571 41130 95 6 0 134237 0
[pid=7412] ppid=7408 vsize=72312 CPUtime=6.28 cores=2
/proc/7412/stat : 7412 (lingeling) R 7408 7408 7235 0 -1 4202496 32287 0 0 0 622 6 0 0 25 0 1 0 514589950 74047488 17244 33554432000 4194304 4352236 140737120871168 18446744073709551615 4267768 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/7412/statm: 18078 17244 125 39 0 17158 0
[pid=7413] ppid=7408 vsize=196608 CPUtime=3.1 cores=6
/proc/7413/stat : 7413 (march_hi) R 7408 7408 7235 0 -1 4194304 84145 0 0 0 299 11 0 0 25 0 1 0 514589950 201326592 34394 33554432000 4194304 4331988 140734996688624 18446744073709551615 4251649 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/7413/statm: 49152 34394 129 34 0 47594 0
Current children cumulated CPU time (s) 25.1
Current children cumulated vsize (KiB) 1172872

[startup+12.7009 s]
/proc/loadavg: 9.79 9.92 7.37 11/188 7414
/proc/meminfo: memFree=13385120/32951124 swapFree=67111528/67111528
[pid=7408] ppid=7406 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/7408/stat : 7408 (ppfolio) S 7406 7408 7235 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 514589950 18763776 390 33554432000 4194304 4287625 140736069988608 18446744073709551615 244446965263 0 0 4096 65536 6965881017840173055 0 0 17 4 0 0 0
/proc/7408/statm: 4581 390 341 23 0 91 0
[pid=7409] ppid=7408 vsize=308248 CPUtime=12.69 cores=0
/proc/7409/stat : 7409 (cryptominisat) R 7408 7408 7235 0 -1 4202496 67474 0 0 0 1258 11 0 0 25 0 1 0 514589950 315645952 61028 33554432000 4194304 4587354 140733601901696 18446744073709551615 4237467 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/7409/statm: 77062 61028 330 96 0 73624 0
[pid=7410] ppid=7408 vsize=131928 CPUtime=12.69 cores=4
/proc/7410/stat : 7410 (clasp) R 7408 7408 7235 0 -1 4202496 44573 0 0 0 1264 5 0 0 25 0 1 0 514589950 135094272 29540 33554432000 4194304 4785740 140734651353680 18446744073709551615 4553288 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/7410/statm: 32982 29540 340 145 0 30027 0
[pid=7411] ppid=7408 vsize=542284 CPUtime=6.31 cores=6
/proc/7411/stat : 7411 (TNM) R 7408 7408 7235 0 -1 4202496 41233 0 0 0 624 7 0 0 25 0 1 0 514589950 555298816 41130 33554432000 134512640 134534564 4292633392 18446744073709551615 134529944 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/7411/statm: 135571 41130 95 6 0 134237 0
[pid=7412] ppid=7408 vsize=55456 CPUtime=12.68 cores=2
/proc/7412/stat : 7412 (lingeling) R 7408 7408 7235 0 -1 4202496 32289 0 0 0 1262 6 0 0 25 0 1 0 514589950 56786944 13032 33554432000 4194304 4352236 140737120871168 18446744073709551615 4328100 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/7412/statm: 13864 13032 127 39 0 12944 0
[pid=7413] ppid=7408 vsize=222440 CPUtime=6.3 cores=6
/proc/7413/stat : 7413 (march_hi) R 7408 7408 7235 0 -1 4202496 164054 0 0 0 609 21 0 0 25 0 1 0 514589950 227778560 41148 33554432000 4194304 4331988 140734996688624 18446744073709551615 244446594440 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/7413/statm: 55610 41148 150 34 0 54052 0
Current children cumulated CPU time (s) 50.67
Current children cumulated vsize (KiB) 1278680

[startup+25.5006 s]
/proc/loadavg: 9.91 9.94 7.42 11/188 7414
/proc/meminfo: memFree=13250580/32951124 swapFree=67111528/67111528
[pid=7408] ppid=7406 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/7408/stat : 7408 (ppfolio) S 7406 7408 7235 0 -1 4202496 523 0 0 0 0 0 0 0 15 0 1 0 514589950 18763776 390 33554432000 4194304 4287625 140736069988608 18446744073709551615 244446965263 0 0 4096 65536 6965881017840173055 0 0 17 4 0 0 0
/proc/7408/statm: 4581 390 341 23 0 91 0
[pid=7409] ppid=7408 vsize=1876252 CPUtime=25.49 cores=0
/proc/7409/stat : 7409 (cryptominisat) R 7408 7408 7235 0 -1 4202496 174316 0 0 0 2523 26 0 0 25 0 1 0 514589950 1921282048 55736 33554432000 4194304 4587354 140733601901696 18446744073709551615 4237679 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/7409/statm: 469063 55736 335 96 0 465625 0
[pid=7410] ppid=7408 vsize=139564 CPUtime=25.49 cores=4
/proc/7410/stat : 7410 (clasp) R 7408 7408 7235 0 -1 4202496 46898 0 0 0 2543 6 0 0 25 0 1 0 514589950 142913536 31865 33554432000 4194304 4785740 140734651353680 18446744073709551615 4553283 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/7410/statm: 34891 31865 340 145 0 31936 0
[pid=7411] ppid=7408 vsize=542284 CPUtime=12.64 cores=6
/proc/7411/stat : 7411 (TNM) R 7408 7408 7235 0 -1 4202496 41233 0 0 0 1257 7 0 0 25 0 1 0 514589950 555298816 41130 33554432000 134512640 134534564 4292633392 18446744073709551615 134529366 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/7411/statm: 135571 41130 95 6 0 134237 0
[pid=7412] ppid=7408 vsize=37252 CPUtime=25.48 cores=2
/proc/7412/stat : 7412 (lingeling) R 7408 7408 7235 0 -1 4202496 32289 0 0 0 2542 6 0 0 25 0 1 0 514589950 38146048 8481 33554432000 4194304 4352236 140737120871168 18446744073709551615 4233979 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/7412/statm: 9313 8481 127 39 0 8393 0
[pid=7413] ppid=7408 vsize=598892 CPUtime=12.7 cores=6
/proc/7413/stat : 7413 (march_hi) R 7408 7408 7235 0 -1 4202496 248116 0 0 0 1238 32 0 0 25 0 1 0 514589950 613265408 69035 33554432000 4194304 4331988 140734996688624 18446744073709551615 4210516 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/7413/statm: 149723 69035 172 34 0 148165 0
Current children cumulated CPU time (s) 101.8
Current children cumulated vsize (KiB) 3212568

[startup+51.1008 s]
/proc/loadavg: 10.01 9.96 7.49 11/188 7415
/proc/meminfo: memFree=13167140/32951124 swapFree=67111528/67111528
[pid=7408] ppid=7406 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/7408/stat : 7408 (ppfolio) S 7406 7408 7235 0 -1 4202496 523 0 0 0 0 0 0 0 15 0 1 0 514589950 18763776 390 33554432000 4194304 4287625 140736069988608 18446744073709551615 244446965263 0 0 4096 65536 6965881017840173055 0 0 17 4 0 0 0
/proc/7408/statm: 4581 390 341 23 0 91 0
[pid=7409] ppid=7408 vsize=2037384 CPUtime=51.09 cores=0
/proc/7409/stat : 7409 (cryptominisat) R 7408 7408 7235 0 -1 4202496 177870 0 0 0 5082 27 0 0 25 0 1 0 514589950 2086281216 59289 33554432000 4194304 4587354 140733601901696 18446744073709551615 4238658 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/7409/statm: 509346 59289 340 96 0 505908 0
[pid=7410] ppid=7408 vsize=187240 CPUtime=51.09 cores=4
/proc/7410/stat : 7410 (clasp) R 7408 7408 7235 0 -1 4202496 58622 0 0 0 5098 11 0 0 25 0 1 0 514589950 191733760 43589 33554432000 4194304 4785740 140734651353680 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/7410/statm: 46810 43589 340 145 0 43855 0
[pid=7411] ppid=7408 vsize=542284 CPUtime=25.41 cores=6
/proc/7411/stat : 7411 (TNM) R 7408 7408 7235 0 -1 4202496 41233 0 0 0 2534 7 0 0 25 0 1 0 514589950 555298816 41130 33554432000 134512640 134534564 4292633392 18446744073709551615 134525956 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/7411/statm: 135571 41130 95 6 0 134237 0
[pid=7412] ppid=7408 vsize=37188 CPUtime=51.07 cores=2

################
# More data... #
################

/proc/7413/statm: 160142 80500 175 34 0 158584 0
Current children cumulated CPU time (s) 3763.98
Current children cumulated vsize (KiB) 3560264

[startup+1002.3 s]
/proc/loadavg: 10.24 10.17 9.19 11/188 7472
/proc/meminfo: memFree=13326152/32951124 swapFree=67111528/67111528
[pid=7408] ppid=7406 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/7408/stat : 7408 (ppfolio) S 7406 7408 7235 0 -1 4202496 523 0 0 0 0 0 0 0 15 0 1 0 514589950 18763776 390 33554432000 4194304 4287625 140736069988608 18446744073709551615 244446965263 0 0 4096 65536 6965881017840173055 0 0 17 4 0 0 0
/proc/7408/statm: 4581 390 341 23 0 91 0
[pid=7409] ppid=7408 vsize=2107808 CPUtime=1002.29 cores=0
/proc/7409/stat : 7409 (cryptominisat) R 7408 7408 7235 0 -1 4202496 363011 0 0 0 100136 93 0 0 25 0 1 0 514589950 2158395392 64722 33554432000 4194304 4587354 140733601901696 18446744073709551615 4238314 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/7409/statm: 526952 64722 341 96 0 523514 0
[pid=7410] ppid=7408 vsize=455404 CPUtime=1002.32 cores=4
/proc/7410/stat : 7410 (clasp) R 7408 7408 7235 0 -1 4202496 124616 0 0 0 100196 36 0 0 25 0 1 0 514589950 466333696 109583 33554432000 4194304 4785740 140734651353680 18446744073709551615 4287028 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/7410/statm: 113851 109583 340 145 0 110896 0
[pid=7411] ppid=7408 vsize=542284 CPUtime=498.42 cores=6
/proc/7411/stat : 7411 (TNM) R 7408 7408 7235 0 -1 4202496 41233 0 0 0 49835 7 0 0 25 0 1 0 514589950 555298816 41130 33554432000 134512640 134534564 4292633392 18446744073709551615 134528511 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/7411/statm: 135571 41130 95 6 0 134237 0
[pid=7412] ppid=7408 vsize=73656 CPUtime=1002.19 cores=2
/proc/7412/stat : 7412 (lingeling) R 7408 7408 7235 0 -1 4202496 65086 0 0 0 100207 12 0 0 25 0 1 0 514589950 75423744 17582 33554432000 4194304 4352236 140737120871168 18446744073709551615 4328068 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/7412/statm: 18414 17582 127 39 0 17494 0
[pid=7413] ppid=7408 vsize=641176 CPUtime=498.43 cores=6
/proc/7413/stat : 7413 (march_hi) R 7408 7408 7235 0 -1 4202496 259760 0 0 0 49808 35 0 0 25 0 1 0 514589950 656564224 80679 33554432000 4194304 4331988 140734996688624 18446744073709551615 4265323 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/7413/statm: 160294 80679 175 34 0 158736 0
Current children cumulated CPU time (s) 4003.65
Current children cumulated vsize (KiB) 3838652

[startup+1062.3 s]
/proc/loadavg: 10.08 10.14 9.23 11/188 7474
/proc/meminfo: memFree=13286700/32951124 swapFree=67111528/67111528
[pid=7408] ppid=7406 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/7408/stat : 7408 (ppfolio) S 7406 7408 7235 0 -1 4202496 523 0 0 0 0 0 0 0 15 0 1 0 514589950 18763776 390 33554432000 4194304 4287625 140736069988608 18446744073709551615 244446965263 0 0 4096 65536 6965881017840173055 0 0 17 4 0 0 0
/proc/7408/statm: 4581 390 341 23 0 91 0
[pid=7409] ppid=7408 vsize=1967708 CPUtime=1062.29 cores=0
/proc/7409/stat : 7409 (cryptominisat) R 7408 7408 7235 0 -1 4202496 376964 0 0 0 106132 97 0 0 25 0 1 0 514589950 2014932992 72991 33554432000 4194304 4587354 140733601901696 18446744073709551615 4357668 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/7409/statm: 491927 72991 341 96 0 488489 0
[pid=7410] ppid=7408 vsize=456364 CPUtime=1062.32 cores=4
/proc/7410/stat : 7410 (clasp) R 7408 7408 7235 0 -1 4202496 124853 0 0 0 106196 36 0 0 25 0 1 0 514589950 467316736 109820 33554432000 4194304 4785740 140734651353680 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/7410/statm: 114091 109820 340 145 0 111136 0
[pid=7411] ppid=7408 vsize=542284 CPUtime=528.26 cores=6
/proc/7411/stat : 7411 (TNM) R 7408 7408 7235 0 -1 4202496 41233 0 0 0 52819 7 0 0 25 0 1 0 514589950 555298816 41130 33554432000 134512640 134534564 4292633392 18446744073709551615 134525938 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/7411/statm: 135571 41130 95 6 0 134237 0
[pid=7412] ppid=7408 vsize=73656 CPUtime=1062.18 cores=2
/proc/7412/stat : 7412 (lingeling) R 7408 7408 7235 0 -1 4202496 67134 0 0 0 106206 12 0 0 25 0 1 0 514589950 75423744 17582 33554432000 4194304 4352236 140737120871168 18446744073709551615 4328194 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/7412/statm: 18414 17582 127 39 0 17494 0
[pid=7413] ppid=7408 vsize=641464 CPUtime=528.26 cores=6
/proc/7413/stat : 7413 (march_hi) R 7408 7408 7235 0 -1 4202496 259842 0 0 0 52791 35 0 0 25 0 1 0 514589950 656859136 80761 33554432000 4194304 4331988 140734996688624 18446744073709551615 4263309 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/7413/statm: 160366 80761 175 34 0 158808 0
Current children cumulated CPU time (s) 4243.31
Current children cumulated vsize (KiB) 3699800

[startup+1122.3 s]
/proc/loadavg: 10.13 10.14 9.29 11/188 7476
/proc/meminfo: memFree=13274148/32951124 swapFree=67111528/67111528
[pid=7408] ppid=7406 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/7408/stat : 7408 (ppfolio) S 7406 7408 7235 0 -1 4202496 523 0 0 0 0 0 0 0 15 0 1 0 514589950 18763776 390 33554432000 4194304 4287625 140736069988608 18446744073709551615 244446965263 0 0 4096 65536 6965881017840173055 0 0 17 4 0 0 0
/proc/7408/statm: 4581 390 341 23 0 91 0
[pid=7409] ppid=7408 vsize=1870772 CPUtime=1122.29 cores=0
/proc/7409/stat : 7409 (cryptominisat) R 7408 7408 7235 0 -1 4202496 381878 0 0 0 112130 99 0 0 25 0 1 0 514589950 1915670528 71836 33554432000 4194304 4587354 140733601901696 18446744073709551615 4243539 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/7409/statm: 467693 71836 341 96 0 464255 0
[pid=7410] ppid=7408 vsize=472988 CPUtime=1122.32 cores=4
/proc/7410/stat : 7410 (clasp) R 7408 7408 7235 0 -1 4202496 129005 0 0 0 112195 37 0 0 25 0 1 0 514589950 484339712 113972 33554432000 4194304 4785740 140734651353680 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/7410/statm: 118247 113972 340 145 0 115292 0
[pid=7411] ppid=7408 vsize=542284 CPUtime=558.08 cores=6
/proc/7411/stat : 7411 (TNM) R 7408 7408 7235 0 -1 4202496 41233 0 0 0 55801 7 0 0 25 0 1 0 514589950 555298816 41130 33554432000 134512640 134534564 4292633392 18446744073709551615 134528659 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/7411/statm: 135571 41130 95 6 0 134237 0
[pid=7412] ppid=7408 vsize=73656 CPUtime=1122.18 cores=2
/proc/7412/stat : 7412 (lingeling) R 7408 7408 7235 0 -1 4202496 69182 0 0 0 112205 13 0 0 25 0 1 0 514589950 75423744 17582 33554432000 4194304 4352236 140737120871168 18446744073709551615 4328162 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/7412/statm: 18414 17582 127 39 0 17494 0
[pid=7413] ppid=7408 vsize=641744 CPUtime=558.12 cores=6
/proc/7413/stat : 7413 (march_hi) R 7408 7408 7235 0 -1 4202496 259906 0 0 0 55777 35 0 0 25 0 1 0 514589950 657145856 80825 33554432000 4194304 4331988 140734996688624 18446744073709551615 244446340689 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/7413/statm: 160436 80825 175 34 0 158878 0
Current children cumulated CPU time (s) 4482.99
Current children cumulated vsize (KiB) 3619768

[startup+1182.3 s]
/proc/loadavg: 10.09 10.13 9.34 11/188 7477
/proc/meminfo: memFree=13265192/32951124 swapFree=67111528/67111528
[pid=7408] ppid=7406 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/7408/stat : 7408 (ppfolio) S 7406 7408 7235 0 -1 4202496 523 0 0 0 0 0 0 0 15 0 1 0 514589950 18763776 390 33554432000 4194304 4287625 140736069988608 18446744073709551615 244446965263 0 0 4096 65536 6965881017840173055 0 0 17 4 0 0 0
/proc/7408/statm: 4581 390 341 23 0 91 0
[pid=7409] ppid=7408 vsize=1846276 CPUtime=1182.29 cores=0
/proc/7409/stat : 7409 (cryptominisat) R 7408 7408 7235 0 -1 4202496 387793 0 0 0 118127 102 0 0 25 0 1 0 514589950 1890586624 72005 33554432000 4194304 4587354 140733601901696 18446744073709551615 4238431 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/7409/statm: 461569 72005 341 96 0 458131 0
[pid=7410] ppid=7408 vsize=480160 CPUtime=1182.33 cores=4
/proc/7410/stat : 7410 (clasp) R 7408 7408 7235 0 -1 4202496 130742 0 0 0 118195 38 0 0 25 0 1 0 514589950 491683840 115709 33554432000 4194304 4785740 140734651353680 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/7410/statm: 120040 115709 340 145 0 117085 0
[pid=7411] ppid=7408 vsize=542284 CPUtime=587.95 cores=6
/proc/7411/stat : 7411 (TNM) R 7408 7408 7235 0 -1 4202496 41233 0 0 0 58788 7 0 0 25 0 1 0 514589950 555298816 41130 33554432000 134512640 134534564 4292633392 18446744073709551615 134528957 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/7411/statm: 135571 41130 95 6 0 134237 0
[pid=7412] ppid=7408 vsize=73652 CPUtime=1182.18 cores=2
/proc/7412/stat : 7412 (lingeling) R 7408 7408 7235 0 -1 4202496 69182 0 0 0 118205 13 0 0 25 0 1 0 514589950 75419648 17581 33554432000 4194304 4352236 140737120871168 18446744073709551615 4328345 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/7412/statm: 18413 17581 127 39 0 17493 0
[pid=7413] ppid=7408 vsize=641900 CPUtime=587.92 cores=6
/proc/7413/stat : 7413 (march_hi) R 7408 7408 7235 0 -1 4202496 259946 0 0 0 58757 35 0 0 25 0 1 0 514589950 657305600 80865 33554432000 4194304 4331988 140734996688624 18446744073709551615 4263319 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/7413/statm: 160475 80865 175 34 0 158917 0
Current children cumulated CPU time (s) 4722.67
Current children cumulated vsize (KiB) 3602596



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1201.7 s]
/proc/loadavg: 10.12 10.13 9.35 11/188 7478
/proc/meminfo: memFree=13268284/32951124 swapFree=67111528/67111528
[pid=7408] ppid=7406 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/7408/stat : 7408 (ppfolio) S 7406 7408 7235 0 -1 4202496 523 0 0 0 0 0 0 0 15 0 1 0 514589950 18763776 390 33554432000 4194304 4287625 140736069988608 18446744073709551615 244446965263 0 0 4096 65536 6965881017840173055 0 0 17 4 0 0 0
/proc/7408/statm: 4581 390 341 23 0 91 0
[pid=7409] ppid=7408 vsize=1846276 CPUtime=1201.7 cores=0
/proc/7409/stat : 7409 (cryptominisat) R 7408 7408 7235 0 -1 4202496 388549 0 0 0 120068 102 0 0 25 0 1 0 514589950 1890586624 72747 33554432000 4194304 4587354 140733601901696 18446744073709551615 4238778 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/7409/statm: 461569 72747 341 96 0 458131 0
[pid=7410] ppid=7408 vsize=481440 CPUtime=1201.73 cores=4
/proc/7410/stat : 7410 (clasp) R 7408 7408 7235 0 -1 4202496 131047 0 0 0 120135 38 0 0 25 0 1 0 514589950 492994560 116014 33554432000 4194304 4785740 140734651353680 18446744073709551615 4286953 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/7410/statm: 120360 116014 340 145 0 117405 0
[pid=7411] ppid=7408 vsize=542284 CPUtime=597.56 cores=6
/proc/7411/stat : 7411 (TNM) R 7408 7408 7235 0 -1 4202496 41233 0 0 0 59749 7 0 0 25 0 1 0 514589950 555298816 41130 33554432000 134512640 134534564 4292633392 18446744073709551615 134528659 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/7411/statm: 135571 41130 95 6 0 134237 0
[pid=7412] ppid=7408 vsize=65452 CPUtime=1201.58 cores=2
/proc/7412/stat : 7412 (lingeling) R 7408 7408 7235 0 -1 4202496 69182 0 0 0 120145 13 0 0 25 0 1 0 514589950 67022848 15531 33554432000 4194304 4352236 140737120871168 18446744073709551615 4328095 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/7412/statm: 16363 15531 127 39 0 15443 0
[pid=7413] ppid=7408 vsize=642032 CPUtime=597.61 cores=6
/proc/7413/stat : 7413 (march_hi) R 7408 7408 7235 0 -1 4202496 259955 0 0 0 59726 35 0 0 25 0 1 0 514589950 657440768 80874 33554432000 4194304 4331988 140734996688624 18446744073709551615 4237147 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/7413/statm: 160508 80874 175 34 0 158950 0
Current children cumulated CPU time (s) 4800.18
Current children cumulated vsize (KiB) 3595808

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1201.7 s]
/proc/loadavg: 10.12 10.13 9.35 11/188 7478
/proc/meminfo: memFree=13268284/32951124 swapFree=67111528/67111528
[pid=7408] ppid=7406 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/7408/stat : 7408 (ppfolio) S 7406 7408 7235 0 -1 4202496 523 0 0 0 0 0 0 0 15 0 1 0 514589950 18763776 390 33554432000 4194304 4287625 140736069988608 18446744073709551615 244446965263 0 0 4096 65536 6965881017840173055 0 0 17 4 0 0 0
/proc/7408/statm: 4581 390 341 23 0 91 0
[pid=7409] ppid=7408 vsize=1846276 CPUtime=1201.7 cores=0
/proc/7409/stat : 7409 (cryptominisat) R 7408 7408 7235 0 -1 4202496 388549 0 0 0 120068 102 0 0 25 0 1 0 514589950 1890586624 72747 33554432000 4194304 4587354 140733601901696 18446744073709551615 4238778 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/7409/statm: 461569 72747 341 96 0 458131 0
[pid=7410] ppid=7408 vsize=481440 CPUtime=1201.73 cores=4
/proc/7410/stat : 7410 (clasp) R 7408 7408 7235 0 -1 4202496 131047 0 0 0 120135 38 0 0 25 0 1 0 514589950 492994560 116014 33554432000 4194304 4785740 140734651353680 18446744073709551615 4286953 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/7410/statm: 120360 116014 340 145 0 117405 0
[pid=7411] ppid=7408 vsize=542284 CPUtime=597.56 cores=6
/proc/7411/stat : 7411 (TNM) R 7408 7408 7235 0 -1 4202496 41233 0 0 0 59749 7 0 0 25 0 1 0 514589950 555298816 41130 33554432000 134512640 134534564 4292633392 18446744073709551615 134528659 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/7411/statm: 135571 41130 95 6 0 134237 0
[pid=7412] ppid=7408 vsize=65452 CPUtime=1201.58 cores=2
/proc/7412/stat : 7412 (lingeling) R 7408 7408 7235 0 -1 4202496 69182 0 0 0 120145 13 0 0 25 0 1 0 514589950 67022848 15531 33554432000 4194304 4352236 140737120871168 18446744073709551615 4328095 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/7412/statm: 16363 15531 127 39 0 15443 0
[pid=7413] ppid=7408 vsize=642032 CPUtime=597.61 cores=6
/proc/7413/stat : 7413 (march_hi) R 7408 7408 7235 0 -1 4202496 259955 0 0 0 59726 35 0 0 25 0 1 0 514589950 657440768 80874 33554432000 4194304 4331988 140734996688624 18446744073709551615 4237147 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/7413/statm: 160508 80874 175 34 0 158950 0
Current children cumulated CPU time (s) 4800.18
Current children cumulated vsize (KiB) 3595808

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 7408 and gives
#  childrusage.ru_utime.tv_sec=0
#  childrusage.ru_utime.tv_usec=999
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=999
# CPU time returned by wait4() is 0.001998
# while last known CPU time is 4800.18
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1201.71
CPU time (s): 4800.18
CPU user time (s): 4798.23
CPU system time (s): 1.95
CPU usage (%): 399.444
Max. virtual memory (cumulated for all children) (KiB): 5496896

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 0.000999
system time used= 0.000999
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 523
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 333
involuntary context switches= 6

runsolver used 1.75873 second user time and 4.80527 second system time

The end

Launcher Data 

Begin job on node105 at 2011-04-29 00:31:35
IDJOB=3280290
IDBENCH=20365
IDSOLVER=1612
FILE ID=node105/3280290-1304029894
RUNJOBID= node105-1304028688-7254
PBS_JOBID= 13170347
Free space on /tmp= 71888 MiB

SOLVER NAME= ppfolio par
BENCH NAME= SAT07/industrial/anbulagan/hard-unsat/total-10-17-u.cnf
COMMAND LINE= ppfolio BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3280290-1304029894/watcher-3280290-1304029894 -o /tmp/evaluation-result-3280290-1304029894/solver-3280290-1304029894 -C 4800 -W 1300 -M 15500  ppfolio HOME/instance-3280290-1304029894.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 9ce7c48a28ddd5790e38607bb893dd38
RANDOM SEED=1292810014

node105.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5410.70
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      14150392 kB
Buffers:       2296368 kB
Cached:       14426768 kB
SwapCached:          0 kB
Active:       10374908 kB
Inactive:      7852200 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      14150392 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:           34944 kB
Writeback:           0 kB
AnonPages:     1503264 kB
Mapped:          16720 kB
Slab:           507092 kB
PageTables:       7684 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:  4199176 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71876 MiB
End job on node105 at 2011-04-29 00:51:39