Trace number 3190858

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ppfolio par? (TO) 4800.25 1201.71

General information on the benchmark

Namerandom/large/
unif-k3-r4.2-v35000-c147000-S448503257-098-UNKNOWN.cnf
MD5SUM0d20733a6f7d3b01f88ae7bba5f0d271
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark405.867
Satisfiable
(Un)Satisfiability was proved
Number of variables34999
Number of clauses147000
Sum of the clauses size441000
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 3147000
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.00/0.00	This is pico-portfolio, a naive parallel portfolio
0.00/0.00	Copyright (C) 2011 Olivier Roussel
0.00/0.00	Using 4 processing units
0.00/0.00	This portfolio uses the following solvers:
0.00/0.00	[1] cryptominisat 2.7.1 (Mate Soos)
0.00/0.00	[2] clasp 1.3.6 (Martin Gebser, Benjamin Kaufmann, and Torsten Schaub)
0.00/0.00	[3] TNM 2009 (Wanxia Wei and Chu Min Li)
0.00/0.00	[4] lingeling 276-6264d55-100731 (Armin Biere)
0.00/0.00	[5] march_hi 2009 (Marijn Heule and Hans Van Maaren)
0.00/0.00	Starting cryptominisat 2.7.1 on core(s) 0
0.00/0.00	Starting clasp 1.3.6 on core(s) 4
0.00/0.00	Starting TNM 2009 on core(s) 6
0.00/0.00	Starting lingeling 276-6264d55-100731 on core(s) 2
0.00/0.00	Starting march_hi 2009 on core(s) 6

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 977)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3190858-1307633702/watcher-3190858-1307633702 -o /tmp/evaluation-result-3190858-1307633702/solver-3190858-1307633702 -C 4800 -W 1300 -M 15500 ppfolio HOME/instance-3190858-1307633702.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 9.16 6.93 3.33 6/181 1177
/proc/meminfo: memFree=29841444/32951124 swapFree=67050508/67111528
[pid=1177] ppid=1175 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/1177/stat : 1177 (ppfolio) S 1175 1177 1071 0 -1 4202496 524 0 0 0 0 0 0 0 19 0 1 0 874972603 18763776 390 33554432000 4194304 4287625 140735567303168 18446744073709551615 212436037135 0 0 4096 65536 4585439030899900415 0 0 17 4 0 0 0
/proc/1177/statm: 4581 390 341 23 0 91 0

[startup+1.02058 s]
/proc/loadavg: 9.16 6.93 3.33 6/181 1177
/proc/meminfo: memFree=29841444/32951124 swapFree=67050508/67111528
[pid=1177] ppid=1175 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/1177/stat : 1177 (ppfolio) S 1175 1177 1071 0 -1 4202496 524 0 0 0 0 0 0 0 19 0 1 0 874972603 18763776 390 33554432000 4194304 4287625 140735567303168 18446744073709551615 212436037135 0 0 4096 65536 4585439030899900415 0 0 17 4 0 0 0
/proc/1177/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+1.12056 s]
/proc/loadavg: 9.16 6.93 3.33 6/181 1177
/proc/meminfo: memFree=29841444/32951124 swapFree=67050508/67111528
[pid=1177] ppid=1175 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/1177/stat : 1177 (ppfolio) S 1175 1177 1071 0 -1 4202496 524 0 0 0 0 0 0 0 19 0 1 0 874972603 18763776 390 33554432000 4194304 4287625 140735567303168 18446744073709551615 212436037135 0 0 4096 65536 4585439030899900415 0 0 17 4 0 0 0
/proc/1177/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+1.20055 s]
/proc/loadavg: 9.16 6.93 3.33 6/181 1177
/proc/meminfo: memFree=29841444/32951124 swapFree=67050508/67111528
[pid=1177] ppid=1175 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/1177/stat : 1177 (ppfolio) S 1175 1177 1071 0 -1 4202496 524 0 0 0 0 0 0 0 19 0 1 0 874972603 18763776 390 33554432000 4194304 4287625 140735567303168 18446744073709551615 212436037135 0 0 4096 65536 4585439030899900415 0 0 17 4 0 0 0
/proc/1177/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+1.32053 s]
/proc/loadavg: 9.16 6.93 3.33 6/181 1177
/proc/meminfo: memFree=29841444/32951124 swapFree=67050508/67111528
[pid=1177] ppid=1175 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/1177/stat : 1177 (ppfolio) S 1175 1177 1071 0 -1 4202496 524 0 0 0 0 0 0 0 19 0 1 0 874972603 18763776 390 33554432000 4194304 4287625 140735567303168 18446744073709551615 212436037135 0 0 4096 65536 4585439030899900415 0 0 17 4 0 0 0
/proc/1177/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+1.52049 s]
/proc/loadavg: 9.16 6.93 3.33 6/181 1177
/proc/meminfo: memFree=29841444/32951124 swapFree=67050508/67111528
[pid=1177] ppid=1175 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/1177/stat : 1177 (ppfolio) S 1175 1177 1071 0 -1 4202496 524 0 0 0 0 0 0 0 19 0 1 0 874972603 18763776 390 33554432000 4194304 4287625 140735567303168 18446744073709551615 212436037135 0 0 4096 65536 4585439030899900415 0 0 17 4 0 0 0
/proc/1177/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+3.10119 s]
/proc/loadavg: 9.23 6.99 3.37 10/187 1183
/proc/meminfo: memFree=29721848/32951124 swapFree=67050508/67111528
[pid=1177] ppid=1175 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/1177/stat : 1177 (ppfolio) S 1175 1177 1071 0 -1 4202496 524 0 0 0 0 0 0 0 18 0 1 0 874972603 18763776 390 33554432000 4194304 4287625 140735567303168 18446744073709551615 212436037135 0 0 4096 65536 4585439030899900415 0 0 17 4 0 0 0
/proc/1177/statm: 4581 390 341 23 0 91 0
[pid=1178] ppid=1177 vsize=94708 CPUtime=3.09 cores=0
/proc/1178/stat : 1178 (cryptominisat) R 1177 1177 1071 0 -1 4202496 13216 0 0 0 307 2 0 0 25 0 1 0 874972603 96980992 10415 33554432000 4194304 4587354 140737401424432 18446744073709551615 212435713803 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/1178/statm: 23677 10415 326 96 0 20239 0
[pid=1179] ppid=1177 vsize=34632 CPUtime=3.09 cores=4
/proc/1179/stat : 1179 (clasp) R 1177 1177 1071 0 -1 4202496 7839 0 0 0 309 0 0 0 25 0 1 0 874972603 35463168 5928 33554432000 4194304 4785740 140734171852512 18446744073709551615 212435677893 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/1179/statm: 8658 5928 340 145 0 5703 0
[pid=1180] ppid=1177 vsize=465988 CPUtime=1.98 cores=6
/proc/1180/stat : 1180 (TNM) R 1177 1177 1071 0 -1 4202496 5913 0 0 0 198 0 0 0 21 0 1 0 874972604 477171712 5810 33554432000 134512640 134534564 4292329456 18446744073709551615 134528737 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/1180/statm: 116497 5810 95 6 0 115163 0
[pid=1181] ppid=1177 vsize=19296 CPUtime=3.09 cores=2
/proc/1181/stat : 1181 (lingeling) R 1177 1177 1071 0 -1 4202496 6362 0 0 0 308 1 0 0 25 0 1 0 874972604 19759104 3479 33554432000 4194304 4352236 140736751730944 18446744073709551615 212436041623 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/1181/statm: 4824 3479 127 39 0 3904 0
[pid=1182] ppid=1177 vsize=27232 CPUtime=1.1 cores=6
/proc/1182/stat : 1182 (march_hi) R 1177 1177 1071 0 -1 4202496 24663 0 0 0 106 4 0 0 25 0 1 0 874972604 27885568 3947 33554432000 4194304 4331988 140737095286064 18446744073709551615 212436042858 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/1182/statm: 6808 3947 141 34 0 5250 0
Current children cumulated CPU time (s) 12.35
Current children cumulated vsize (KiB) 660180

[startup+6.30059 s]
/proc/loadavg: 9.23 6.99 3.37 10/187 1183
/proc/meminfo: memFree=29678696/32951124 swapFree=67050508/67111528
[pid=1177] ppid=1175 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/1177/stat : 1177 (ppfolio) S 1175 1177 1071 0 -1 4202496 524 0 0 0 0 0 0 0 15 0 1 0 874972603 18763776 390 33554432000 4194304 4287625 140735567303168 18446744073709551615 212436037135 0 0 4096 65536 4585439030899900415 0 0 17 4 0 0 0
/proc/1177/statm: 4581 390 341 23 0 91 0
[pid=1178] ppid=1177 vsize=94708 CPUtime=6.29 cores=0
/proc/1178/stat : 1178 (cryptominisat) R 1177 1177 1071 0 -1 4202496 14911 0 0 0 626 3 0 0 25 0 1 0 874972603 96980992 12104 33554432000 4194304 4587354 140737401424432 18446744073709551615 4236573 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/1178/statm: 23677 12104 326 96 0 20239 0
[pid=1179] ppid=1177 vsize=49532 CPUtime=6.29 cores=4
/proc/1179/stat : 1179 (clasp) R 1177 1177 1071 0 -1 4202496 11531 0 0 0 627 2 0 0 25 0 1 0 874972603 50720768 9620 33554432000 4194304 4785740 140734171852512 18446744073709551615 4553402 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/1179/statm: 12383 9620 340 145 0 9428 0
[pid=1180] ppid=1177 vsize=465988 CPUtime=3.56 cores=6
/proc/1180/stat : 1180 (TNM) R 1177 1177 1071 0 -1 4202496 5913 0 0 0 356 0 0 0 25 0 1 0 874972604 477171712 5810 33554432000 134512640 134534564 4292329456 18446744073709551615 134530606 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/1180/statm: 116497 5810 95 6 0 115163 0
[pid=1181] ppid=1177 vsize=19296 CPUtime=6.29 cores=2
/proc/1181/stat : 1181 (lingeling) R 1177 1177 1071 0 -1 4202496 6362 0 0 0 628 1 0 0 25 0 1 0 874972604 19759104 3479 33554432000 4194304 4352236 140736751730944 18446744073709551615 4328026 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/1181/statm: 4824 3479 127 39 0 3904 0
[pid=1182] ppid=1177 vsize=76464 CPUtime=2.69 cores=6
/proc/1182/stat : 1182 (march_hi) R 1177 1177 1071 0 -1 4202496 52411 0 0 0 262 7 0 0 25 0 1 0 874972604 78299136 15862 33554432000 4194304 4331988 140737095286064 18446744073709551615 4228970 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/1182/statm: 19116 15862 173 34 0 17558 0
Current children cumulated CPU time (s) 25.12
Current children cumulated vsize (KiB) 724312

[startup+12.7004 s]
/proc/loadavg: 9.28 7.07 3.44 10/187 1183
/proc/meminfo: memFree=29624916/32951124 swapFree=67050508/67111528
[pid=1177] ppid=1175 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/1177/stat : 1177 (ppfolio) S 1175 1177 1071 0 -1 4202496 524 0 0 0 0 0 0 0 15 0 1 0 874972603 18763776 390 33554432000 4194304 4287625 140735567303168 18446744073709551615 212436037135 0 0 4096 65536 4585439030899900415 0 0 17 4 0 0 0
/proc/1177/statm: 4581 390 341 23 0 91 0
[pid=1178] ppid=1177 vsize=1515860 CPUtime=12.69 cores=0
/proc/1178/stat : 1178 (cryptominisat) R 1177 1177 1071 0 -1 4202496 19839 0 0 0 1264 5 0 0 25 0 1 0 874972603 1552240640 10699 33554432000 4194304 4587354 140737401424432 18446744073709551615 4518654 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/1178/statm: 378965 10699 331 96 0 375527 0
[pid=1179] ppid=1177 vsize=71232 CPUtime=12.68 cores=4
/proc/1179/stat : 1179 (clasp) R 1177 1177 1071 0 -1 4202496 16948 0 0 0 1264 4 0 0 25 0 1 0 874972603 72941568 15037 33554432000 4194304 4785740 140734171852512 18446744073709551615 4553003 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/1179/statm: 17808 15037 340 145 0 14853 0
[pid=1180] ppid=1177 vsize=465988 CPUtime=6.72 cores=6
/proc/1180/stat : 1180 (TNM) R 1177 1177 1071 0 -1 4202496 5913 0 0 0 672 0 0 0 25 0 1 0 874972604 477171712 5810 33554432000 134512640 134534564 4292329456 18446744073709551615 134530573 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/1180/statm: 116497 5810 95 6 0 115163 0
[pid=1181] ppid=1177 vsize=23448 CPUtime=12.69 cores=2
/proc/1181/stat : 1181 (lingeling) R 1177 1177 1071 0 -1 4202496 7911 0 0 0 1268 1 0 0 25 0 1 0 874972604 24010752 5028 33554432000 4194304 4352236 140736751730944 18446744073709551615 4328055 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/1181/statm: 5862 5028 127 39 0 4942 0
[pid=1182] ppid=1177 vsize=78128 CPUtime=5.89 cores=6
/proc/1182/stat : 1182 (march_hi) R 1177 1177 1071 0 -1 4202496 52830 0 0 0 582 7 0 0 25 0 1 0 874972604 80003072 16281 33554432000 4194304 4331988 140737095286064 18446744073709551615 4228983 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/1182/statm: 19532 16281 173 34 0 17974 0
Current children cumulated CPU time (s) 50.67
Current children cumulated vsize (KiB) 2172980

[startup+25.501 s]
/proc/loadavg: 9.31 7.15 3.50 10/187 1184
/proc/meminfo: memFree=29535728/32951124 swapFree=67050508/67111528
[pid=1177] ppid=1175 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/1177/stat : 1177 (ppfolio) S 1175 1177 1071 0 -1 4202496 524 0 0 0 0 0 0 0 15 0 1 0 874972603 18763776 390 33554432000 4194304 4287625 140735567303168 18446744073709551615 212436037135 0 0 4096 65536 4585439030899900415 0 0 17 4 0 0 0
/proc/1177/statm: 4581 390 341 23 0 91 0
[pid=1178] ppid=1177 vsize=66824 CPUtime=25.49 cores=0
/proc/1178/stat : 1178 (cryptominisat) R 1177 1177 1071 0 -1 4202496 48758 0 0 0 2538 11 0 0 25 0 1 0 874972603 68427776 10254 33554432000 4194304 4587354 140737401424432 18446744073709551615 4238467 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/1178/statm: 16706 10254 337 96 0 13268 0
[pid=1179] ppid=1177 vsize=110300 CPUtime=25.49 cores=4
/proc/1179/stat : 1179 (clasp) R 1177 1177 1071 0 -1 4202496 26690 0 0 0 2541 8 0 0 25 0 1 0 874972603 112947200 24779 33554432000 4194304 4785740 140734171852512 18446744073709551615 4299831 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/1179/statm: 27575 24779 340 145 0 24620 0
[pid=1180] ppid=1177 vsize=465988 CPUtime=13.1 cores=6
/proc/1180/stat : 1180 (TNM) R 1177 1177 1071 0 -1 4202496 5913 0 0 0 1310 0 0 0 25 0 1 0 874972604 477171712 5810 33554432000 134512640 134534564 4292329456 18446744073709551615 134530158 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/1180/statm: 116497 5810 95 6 0 115163 0
[pid=1181] ppid=1177 vsize=24972 CPUtime=25.49 cores=2
/proc/1181/stat : 1181 (lingeling) R 1177 1177 1071 0 -1 4202496 8292 0 0 0 2548 1 0 0 25 0 1 0 874972604 25571328 5409 33554432000 4194304 4352236 140736751730944 18446744073709551615 4328022 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/1181/statm: 6243 5409 127 39 0 5323 0
[pid=1182] ppid=1177 vsize=84748 CPUtime=12.24 cores=6
/proc/1182/stat : 1182 (march_hi) R 1177 1177 1071 0 -1 4202496 54468 0 0 0 1216 8 0 0 25 0 1 0 874972604 86781952 17919 33554432000 4194304 4331988 140737095286064 18446744073709551615 4211493 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/1182/statm: 21187 17919 173 34 0 19629 0
Current children cumulated CPU time (s) 101.81
Current children cumulated vsize (KiB) 771156

[startup+51.1011 s]
/proc/loadavg: 9.20 7.29 3.64 10/187 1185
/proc/meminfo: memFree=29428152/32951124 swapFree=67050508/67111528
[pid=1177] ppid=1175 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/1177/stat : 1177 (ppfolio) S 1175 1177 1071 0 -1 4202496 524 0 0 0 0 0 0 0 15 0 1 0 874972603 18763776 390 33554432000 4194304 4287625 140735567303168 18446744073709551615 212436037135 0 0 4096 65536 4585439030899900415 0 0 17 4 0 0 0
/proc/1177/statm: 4581 390 341 23 0 91 0
[pid=1178] ppid=1177 vsize=148188 CPUtime=51.09 cores=0
/proc/1178/stat : 1178 (cryptominisat) R 1177 1177 1071 0 -1 4202496 89397 0 0 0 5089 20 0 0 25 0 1 0 874972603 151744512 16457 33554432000 4194304 4587354 140737401424432 18446744073709551615 4237679 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/1178/statm: 37047 16457 337 96 0 33609 0
[pid=1179] ppid=1177 vsize=136336 CPUtime=51.08 cores=4
/proc/1179/stat : 1179 (clasp) R 1177 1177 1071 0 -1 4202496 33059 0 0 0 5098 10 0 0 25 0 1 0 874972603 139608064 31148 33554432000 4194304 4785740 140734171852512 18446744073709551615 4553003 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/1179/statm: 34084 31148 340 145 0 31129 0
[pid=1180] ppid=1177 vsize=465988 CPUtime=25.82 cores=6
/proc/1180/stat : 1180 (TNM) R 1177 1177 1071 0 -1 4202496 5913 0 0 0 2582 0 0 0 25 0 1 0 874972604 477171712 5810 33554432000 134512640 134534564 4292329456 18446744073709551615 134528302 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/1180/statm: 116497 5810 95 6 0 115163 0
[pid=1181] ppid=1177 vsize=27516 CPUtime=51.09 cores=2
/proc/1181/stat : 1181 (lingeling) R 1177 1177 1071 0 -1 4202496 8928 0 0 0 5108 1 0 0 25 0 1 0 874972604 28176384 6045 33554432000 4194304 4352236 140736751730944 18446744073709551615 4329060 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/1181/statm: 6879 6045 127 39 0 5959 0
[pid=1182] ppid=1177 vsize=86596 CPUtime=24.99 cores=6
/proc/1182/stat : 1182 (march_hi) R 1177 1177 1071 0 -1 4202496 54960 0 0 0 2490 9 0 0 25 0 1 0 874972604 88674304 18411 33554432000 4194304 4331988 140737095286064 18446744073709551615 4240796 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/1182/statm: 21649 18411 173 34 0 20091 0
Current children cumulated CPU time (s) 204.07
Current children cumulated vsize (KiB) 882948

[startup+102.3 s]
/proc/loadavg: 9.18 7.61 3.96 10/187 1186
/proc/meminfo: memFree=29314388/32951124 swapFree=67050508/67111528
[pid=1177] ppid=1175 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/1177/stat : 1177 (ppfolio) S 1175 1177 1071 0 -1 4202496 524 0 0 0 0 0 0 0 15 0 1 0 874972603 18763776 390 33554432000 4194304 4287625 140735567303168 18446744073709551615 212436037135 0 0 4096 65536 4585439030899900415 0 0 17 4 0 0 0
/proc/1177/statm: 4581 390 341 23 0 91 0
[pid=1178] ppid=1177 vsize=1575264 CPUtime=102.29 cores=0

################
# More data... #
################

/proc/1182/statm: 22969 19716 173 34 0 21411 0
Current children cumulated CPU time (s) 3764.06
Current children cumulated vsize (KiB) 2579448

[startup+1002.3 s]
/proc/loadavg: 10.13 9.54 7.34 11/188 1383
/proc/meminfo: memFree=31388484/32951124 swapFree=67050508/67111528
[pid=1177] ppid=1175 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/1177/stat : 1177 (ppfolio) S 1175 1177 1071 0 -1 4202496 524 0 0 0 0 0 0 0 15 0 1 0 874972603 18763776 390 33554432000 4194304 4287625 140735567303168 18446744073709551615 212436037135 0 0 4096 65536 4585439030899900415 0 0 17 4 0 0 0
/proc/1177/statm: 4581 390 341 23 0 91 0
[pid=1178] ppid=1177 vsize=89092 CPUtime=1002.32 cores=0
/proc/1178/stat : 1178 (cryptominisat) R 1177 1177 1071 0 -1 4202496 609848 0 0 0 100036 196 0 0 25 0 1 0 874972603 91230208 18437 33554432000 4194304 4587354 140737401424432 18446744073709551615 4237467 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/1178/statm: 22273 18437 337 96 0 18835 0
[pid=1179] ppid=1177 vsize=401816 CPUtime=1002.22 cores=4
/proc/1179/stat : 1179 (clasp) R 1177 1177 1071 0 -1 4202496 98284 0 0 0 100191 31 0 0 25 0 1 0 874972603 411459584 96373 33554432000 4194304 4785740 140734171852512 18446744073709551615 4287204 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/1179/statm: 100454 96373 340 145 0 97499 0
[pid=1180] ppid=1177 vsize=465988 CPUtime=498.84 cores=6
/proc/1180/stat : 1180 (TNM) R 1177 1177 1071 0 -1 4202496 5913 0 0 0 49882 2 0 0 25 0 1 0 874972604 477171712 5810 33554432000 134512640 134534564 4292329456 18446744073709551615 134528726 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/1180/statm: 116497 5810 95 6 0 115163 0
[pid=1181] ppid=1177 vsize=30588 CPUtime=1002.32 cores=2
/proc/1181/stat : 1181 (lingeling) R 1177 1177 1071 0 -1 4202496 9696 0 0 0 100231 1 0 0 25 0 1 0 874972604 31322112 6813 33554432000 4194304 4352236 140736751730944 18446744073709551615 4327902 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/1181/statm: 7647 6813 127 39 0 6727 0
[pid=1182] ppid=1177 vsize=91876 CPUtime=498.02 cores=6
/proc/1182/stat : 1182 (march_hi) R 1177 1177 1071 0 -1 4202496 56295 0 0 0 49790 12 0 0 25 0 1 0 874972604 94081024 19746 33554432000 4194304 4331988 140737095286064 18446744073709551615 4297461 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/1182/statm: 22969 19746 173 34 0 21411 0
Current children cumulated CPU time (s) 4003.72
Current children cumulated vsize (KiB) 1097684

[startup+1062.3 s]
/proc/loadavg: 10.19 9.65 7.52 11/188 1391
/proc/meminfo: memFree=31317348/32951124 swapFree=67050508/67111528
[pid=1177] ppid=1175 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/1177/stat : 1177 (ppfolio) S 1175 1177 1071 0 -1 4202496 524 0 0 0 0 0 0 0 15 0 1 0 874972603 18763776 390 33554432000 4194304 4287625 140735567303168 18446744073709551615 212436037135 0 0 4096 65536 4585439030899900415 0 0 17 4 0 0 0
/proc/1177/statm: 4581 390 341 23 0 91 0
[pid=1178] ppid=1177 vsize=1543304 CPUtime=1062.32 cores=0
/proc/1178/stat : 1178 (cryptominisat) R 1177 1177 1071 0 -1 4202496 630173 0 0 0 106024 208 0 0 25 0 1 0 874972603 1580343296 26394 33554432000 4194304 4587354 140737401424432 18446744073709551615 4238658 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/1178/statm: 385826 26394 337 96 0 382388 0
[pid=1179] ppid=1177 vsize=418184 CPUtime=1062.22 cores=4
/proc/1179/stat : 1179 (clasp) R 1177 1177 1071 0 -1 4202496 102332 0 0 0 106189 33 0 0 25 0 1 0 874972603 428220416 100421 33554432000 4194304 4785740 140734171852512 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/1179/statm: 104546 100421 340 145 0 101591 0
[pid=1180] ppid=1177 vsize=465988 CPUtime=528.72 cores=6
/proc/1180/stat : 1180 (TNM) R 1177 1177 1071 0 -1 4202496 5913 0 0 0 52869 3 0 0 25 0 1 0 874972604 477171712 5810 33554432000 134512640 134534564 4292329456 18446744073709551615 134529080 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/1180/statm: 116497 5810 95 6 0 115163 0
[pid=1181] ppid=1177 vsize=30588 CPUtime=1062.32 cores=2
/proc/1181/stat : 1181 (lingeling) R 1177 1177 1071 0 -1 4202496 9696 0 0 0 106231 1 0 0 25 0 1 0 874972604 31322112 6813 33554432000 4194304 4352236 140736751730944 18446744073709551615 4328026 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/1181/statm: 7647 6813 127 39 0 6727 0
[pid=1182] ppid=1177 vsize=92008 CPUtime=527.82 cores=6
/proc/1182/stat : 1182 (march_hi) R 1177 1177 1071 0 -1 4202496 56324 0 0 0 52770 12 0 0 25 0 1 0 874972604 94216192 19775 33554432000 4194304 4331988 140737095286064 18446744073709551615 4265328 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/1182/statm: 23002 19775 173 34 0 21444 0
Current children cumulated CPU time (s) 4243.4
Current children cumulated vsize (KiB) 2568396

[startup+1122.3 s]
/proc/loadavg: 10.25 9.76 7.69 11/188 1393
/proc/meminfo: memFree=31321148/32951124 swapFree=67050508/67111528
[pid=1177] ppid=1175 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/1177/stat : 1177 (ppfolio) S 1175 1177 1071 0 -1 4202496 524 0 0 0 0 0 0 0 15 0 1 0 874972603 18763776 390 33554432000 4194304 4287625 140735567303168 18446744073709551615 212436037135 0 0 4096 65536 4585439030899900415 0 0 17 4 0 0 0
/proc/1177/statm: 4581 390 341 23 0 91 0
[pid=1178] ppid=1177 vsize=1571204 CPUtime=1122.31 cores=0
/proc/1178/stat : 1178 (cryptominisat) R 1177 1177 1071 0 -1 4202496 648558 0 0 0 112014 217 0 0 25 0 1 0 874972603 1608912896 19488 33554432000 4194304 4587354 140737401424432 18446744073709551615 4230209 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/1178/statm: 392801 19488 337 96 0 389363 0
[pid=1179] ppid=1177 vsize=429824 CPUtime=1122.22 cores=4
/proc/1179/stat : 1179 (clasp) R 1177 1177 1071 0 -1 4202496 105260 0 0 0 112188 34 0 0 25 0 1 0 874972603 440139776 103349 33554432000 4194304 4785740 140734171852512 18446744073709551615 4287204 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/1179/statm: 107456 103349 340 145 0 104501 0
[pid=1180] ppid=1177 vsize=465988 CPUtime=558.52 cores=6
/proc/1180/stat : 1180 (TNM) R 1177 1177 1071 0 -1 4202496 5913 0 0 0 55849 3 0 0 25 0 1 0 874972604 477171712 5810 33554432000 134512640 134534564 4292329456 18446744073709551615 134530593 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/1180/statm: 116497 5810 95 6 0 115163 0
[pid=1181] ppid=1177 vsize=30588 CPUtime=1122.33 cores=2
/proc/1181/stat : 1181 (lingeling) R 1177 1177 1071 0 -1 4202496 9696 0 0 0 112231 2 0 0 25 0 1 0 874972604 31322112 6813 33554432000 4194304 4352236 140736751730944 18446744073709551615 4328100 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/1181/statm: 7647 6813 127 39 0 6727 0
[pid=1182] ppid=1177 vsize=92140 CPUtime=557.7 cores=6
/proc/1182/stat : 1182 (march_hi) R 1177 1177 1071 0 -1 4202496 56352 0 0 0 55758 12 0 0 25 0 1 0 874972604 94351360 19803 33554432000 4194304 4331988 140737095286064 18446744073709551615 4226483 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/1182/statm: 23035 19803 173 34 0 21477 0
Current children cumulated CPU time (s) 4483.08
Current children cumulated vsize (KiB) 2608068

[startup+1182.3 s]
/proc/loadavg: 10.21 9.83 7.84 11/188 1395
/proc/meminfo: memFree=31259960/32951124 swapFree=67050508/67111528
[pid=1177] ppid=1175 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/1177/stat : 1177 (ppfolio) S 1175 1177 1071 0 -1 4202496 524 0 0 0 0 0 0 0 15 0 1 0 874972603 18763776 390 33554432000 4194304 4287625 140735567303168 18446744073709551615 212436037135 0 0 4096 65536 4585439030899900415 0 0 17 4 0 0 0
/proc/1177/statm: 4581 390 341 23 0 91 0
[pid=1178] ppid=1177 vsize=1575472 CPUtime=1182.32 cores=0
/proc/1178/stat : 1178 (cryptominisat) R 1177 1177 1071 0 -1 4202496 669573 0 0 0 118004 228 0 0 25 0 1 0 874972603 1613283328 27384 33554432000 4194304 4587354 140737401424432 18446744073709551615 4230266 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/1178/statm: 393868 27384 337 96 0 390430 0
[pid=1179] ppid=1177 vsize=435152 CPUtime=1182.21 cores=4
/proc/1179/stat : 1179 (clasp) R 1177 1177 1071 0 -1 4202496 106511 0 0 0 118187 34 0 0 25 0 1 0 874972603 445595648 104600 33554432000 4194304 4785740 140734171852512 18446744073709551615 4278587 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/1179/statm: 108788 104600 340 145 0 105833 0
[pid=1180] ppid=1177 vsize=465988 CPUtime=588.38 cores=6
/proc/1180/stat : 1180 (TNM) R 1177 1177 1071 0 -1 4202496 5913 0 0 0 58835 3 0 0 25 0 1 0 874972604 477171712 5810 33554432000 134512640 134534564 4292329456 18446744073709551615 134530568 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/1180/statm: 116497 5810 95 6 0 115163 0
[pid=1181] ppid=1177 vsize=30588 CPUtime=1182.33 cores=2
/proc/1181/stat : 1181 (lingeling) R 1177 1177 1071 0 -1 4202496 9696 0 0 0 118231 2 0 0 25 0 1 0 874972604 31322112 6813 33554432000 4194304 4352236 140736751730944 18446744073709551615 4280960 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/1181/statm: 7647 6813 127 39 0 6727 0
[pid=1182] ppid=1177 vsize=92272 CPUtime=587.51 cores=6
/proc/1182/stat : 1182 (march_hi) R 1177 1177 1071 0 -1 4202496 56377 0 0 0 58739 12 0 0 25 0 1 0 874972604 94486528 19828 33554432000 4194304 4331988 140737095286064 18446744073709551615 4263185 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/1182/statm: 23068 19828 173 34 0 21510 0
Current children cumulated CPU time (s) 4722.75
Current children cumulated vsize (KiB) 2617796



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1201.7 s]
/proc/loadavg: 10.15 9.84 7.88 11/188 1395
/proc/meminfo: memFree=31289948/32951124 swapFree=67050508/67111528
[pid=1177] ppid=1175 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/1177/stat : 1177 (ppfolio) S 1175 1177 1071 0 -1 4202496 524 0 0 0 0 0 0 0 15 0 1 0 874972603 18763776 390 33554432000 4194304 4287625 140735567303168 18446744073709551615 212436037135 0 0 4096 65536 4585439030899900415 0 0 17 4 0 0 0
/proc/1177/statm: 4581 390 341 23 0 91 0
[pid=1178] ppid=1177 vsize=1547924 CPUtime=1201.72 cores=0
/proc/1178/stat : 1178 (cryptominisat) R 1177 1177 1071 0 -1 4202496 674185 0 0 0 119941 231 0 0 25 0 1 0 874972603 1585074176 18607 33554432000 4194304 4587354 140737401424432 18446744073709551615 4356803 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/1178/statm: 386981 18607 337 96 0 383543 0
[pid=1179] ppid=1177 vsize=435152 CPUtime=1201.61 cores=4
/proc/1179/stat : 1179 (clasp) R 1177 1177 1071 0 -1 4202496 106511 0 0 0 120127 34 0 0 25 0 1 0 874972603 445595648 104600 33554432000 4194304 4785740 140734171852512 18446744073709551615 4553068 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/1179/statm: 108788 104600 340 145 0 105833 0
[pid=1180] ppid=1177 vsize=465988 CPUtime=598.01 cores=6
/proc/1180/stat : 1180 (TNM) R 1177 1177 1071 0 -1 4202496 5913 0 0 0 59798 3 0 0 25 0 1 0 874972604 477171712 5810 33554432000 134512640 134534564 4292329456 18446744073709551615 134528877 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/1180/statm: 116497 5810 95 6 0 115163 0
[pid=1181] ppid=1177 vsize=30588 CPUtime=1201.74 cores=2
/proc/1181/stat : 1181 (lingeling) R 1177 1177 1071 0 -1 4202496 9696 0 0 0 120172 2 0 0 25 0 1 0 874972604 31322112 6813 33554432000 4194304 4352236 140736751730944 18446744073709551615 4328026 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/1181/statm: 7647 6813 127 39 0 6727 0
[pid=1182] ppid=1177 vsize=92272 CPUtime=597.17 cores=6
/proc/1182/stat : 1182 (march_hi) R 1177 1177 1071 0 -1 4202496 56385 0 0 0 59705 12 0 0 25 0 1 0 874972604 94486528 19836 33554432000 4194304 4331988 140737095286064 18446744073709551615 4292232 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/1182/statm: 23068 19836 173 34 0 21510 0
Current children cumulated CPU time (s) 4800.25
Current children cumulated vsize (KiB) 2590248

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1201.7 s]
/proc/loadavg: 10.15 9.84 7.88 11/188 1395
/proc/meminfo: memFree=31289948/32951124 swapFree=67050508/67111528
[pid=1177] ppid=1175 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/1177/stat : 1177 (ppfolio) S 1175 1177 1071 0 -1 4202496 524 0 0 0 0 0 0 0 15 0 1 0 874972603 18763776 390 33554432000 4194304 4287625 140735567303168 18446744073709551615 212436037135 0 0 4096 65536 4585439030899900415 0 0 17 4 0 0 0
/proc/1177/statm: 4581 390 341 23 0 91 0
[pid=1178] ppid=1177 vsize=1547924 CPUtime=1201.72 cores=0
/proc/1178/stat : 1178 (cryptominisat) R 1177 1177 1071 0 -1 4202496 674185 0 0 0 119941 231 0 0 25 0 1 0 874972603 1585074176 18607 33554432000 4194304 4587354 140737401424432 18446744073709551615 4356803 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/1178/statm: 386981 18607 337 96 0 383543 0
[pid=1179] ppid=1177 vsize=435152 CPUtime=1201.61 cores=4
/proc/1179/stat : 1179 (clasp) R 1177 1177 1071 0 -1 4202496 106511 0 0 0 120127 34 0 0 25 0 1 0 874972603 445595648 104600 33554432000 4194304 4785740 140734171852512 18446744073709551615 4553068 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/1179/statm: 108788 104600 340 145 0 105833 0
[pid=1180] ppid=1177 vsize=465988 CPUtime=598.01 cores=6
/proc/1180/stat : 1180 (TNM) R 1177 1177 1071 0 -1 4202496 5913 0 0 0 59798 3 0 0 25 0 1 0 874972604 477171712 5810 33554432000 134512640 134534564 4292329456 18446744073709551615 134528877 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/1180/statm: 116497 5810 95 6 0 115163 0
[pid=1181] ppid=1177 vsize=30588 CPUtime=1201.74 cores=2
/proc/1181/stat : 1181 (lingeling) R 1177 1177 1071 0 -1 4202496 9696 0 0 0 120172 2 0 0 25 0 1 0 874972604 31322112 6813 33554432000 4194304 4352236 140736751730944 18446744073709551615 4328026 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/1181/statm: 7647 6813 127 39 0 6727 0
[pid=1182] ppid=1177 vsize=92272 CPUtime=597.17 cores=6
/proc/1182/stat : 1182 (march_hi) R 1177 1177 1071 0 -1 4202496 56385 0 0 0 59705 12 0 0 25 0 1 0 874972604 94486528 19836 33554432000 4194304 4331988 140737095286064 18446744073709551615 4292232 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/1182/statm: 23068 19836 173 34 0 21510 0
Current children cumulated CPU time (s) 4800.25
Current children cumulated vsize (KiB) 2590248

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 1177 and gives
#  childrusage.ru_utime.tv_sec=1201
#  childrusage.ru_utime.tv_usec=724310
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=27995
# CPU time returned by wait4() is 1201.75
# while last known CPU time is 4800.25
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1201.71
CPU time (s): 4800.25
CPU user time (s): 4797.43
CPU system time (s): 2.82
CPU usage (%): 399.451
Max. virtual memory (cumulated for all children) (KiB): 4034484

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1201.72
system time used= 0.027995
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 10222
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 285
involuntary context switches= 3011

runsolver used 1.88371 second user time and 4.67329 second system time

The end

Launcher Data 

Begin job on node102 at 2011-06-09 17:35:02
IDJOB=3190858
IDBENCH=82589
IDSOLVER=1612
FILE ID=node102/3190858-1307633702
RUNJOBID= node102-1307633278-1089
PBS_JOBID= 13549389
Free space on /tmp= 74104 MiB

SOLVER NAME= ppfolio par
BENCH NAME= SAT11/random/large/unif-k3-r4.2-v35000-c147000-S448503257-098-UNKNOWN.cnf
COMMAND LINE= ppfolio BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3190858-1307633702/watcher-3190858-1307633702 -o /tmp/evaluation-result-3190858-1307633702/solver-3190858-1307633702 -C 4800 -W 1300 -M 15500  ppfolio HOME/instance-3190858-1307633702.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 0d20733a6f7d3b01f88ae7bba5f0d271
RANDOM SEED=491775394

node102.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      29841624 kB
Buffers:        202448 kB
Cached:         537064 kB
SwapCached:      25184 kB
Active:        2559364 kB
Inactive:       340996 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      29841624 kB
SwapTotal:    67111528 kB
SwapFree:     67050508 kB
Dirty:           13028 kB
Writeback:           0 kB
AnonPages:     2158212 kB
Mapped:          10472 kB
Slab:           139912 kB
PageTables:       9508 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:  3385200 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 74176 MiB
End job on node102 at 2011-06-09 17:55:06