Trace number 3190929

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ppfolio par? (TO) 4261.63 1300.02

General information on the benchmark

Namerandom/large/
unif-k7-r85-v300-c25500-S69399346-065-UNKNOWN.cnf
MD5SUM98486271f2f72889c040eabfd36ebd13
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables300
Number of clauses25500
Sum of the clauses size178500
Maximum clause length7
Minimum clause length7
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 525500

Solver Data 

0.00/0.00	This is pico-portfolio, a naive parallel portfolio
0.00/0.00	Copyright (C) 2011 Olivier Roussel
0.00/0.00	Using 4 processing units
0.00/0.00	This portfolio uses the following solvers:
0.00/0.00	[1] cryptominisat 2.7.1 (Mate Soos)
0.00/0.00	[2] clasp 1.3.6 (Martin Gebser, Benjamin Kaufmann, and Torsten Schaub)
0.00/0.00	[3] TNM 2009 (Wanxia Wei and Chu Min Li)
0.00/0.00	[4] lingeling 276-6264d55-100731 (Armin Biere)
0.00/0.00	[5] march_hi 2009 (Marijn Heule and Hans Van Maaren)
0.00/0.00	Starting cryptominisat 2.7.1 on core(s) 1
0.00/0.00	Starting clasp 1.3.6 on core(s) 5
0.00/0.00	Starting TNM 2009 on core(s) 7
0.00/0.00	Starting lingeling 276-6264d55-100731 on core(s) 3
0.00/0.00	Starting march_hi 2009 on core(s) 7

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3190929-1304045650/watcher-3190929-1304045650 -o /tmp/evaluation-result-3190929-1304045650/solver-3190929-1304045650 -C 4800 -W 1300 -M 15500 ppfolio HOME/instance-3190929-1304045650.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 3.42 5.37 6.83 9/181 1750
/proc/meminfo: memFree=18268164/32951124 swapFree=67111528/67111528
[pid=1750] ppid=1748 vsize=18324 CPUtime=0 cores=1,3,5,7
/proc/1750/stat : 1750 (ppfolio) S 1748 1750 32317 0 -1 4202496 517 0 0 0 0 0 0 0 19 0 1 0 516165529 18763776 389 33554432000 4194304 4287625 140734631390336 18446744073709551615 230809185807 0 0 4096 65536 2826088480821 0 0 17 1 0 0 0
/proc/1750/statm: 4581 389 341 23 0 91 0

[startup+0.920524 s]
/proc/loadavg: 3.42 5.37 6.83 9/181 1750
/proc/meminfo: memFree=18268164/32951124 swapFree=67111528/67111528
[pid=1750] ppid=1748 vsize=18324 CPUtime=0 cores=1,3,5,7
/proc/1750/stat : 1750 (ppfolio) S 1748 1750 32317 0 -1 4202496 518 0 0 0 0 0 0 0 21 0 1 0 516165529 18763776 389 33554432000 4194304 4287625 140734631390336 18446744073709551615 230809185807 0 0 4096 65536 2962849452177489919 0 0 17 1 0 0 0
/proc/1750/statm: 4581 389 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+1.00051 s]
/proc/loadavg: 3.42 5.37 6.83 9/181 1750
/proc/meminfo: memFree=18268164/32951124 swapFree=67111528/67111528
[pid=1750] ppid=1748 vsize=18324 CPUtime=0 cores=1,3,5,7
/proc/1750/stat : 1750 (ppfolio) S 1748 1750 32317 0 -1 4202496 518 0 0 0 0 0 0 0 21 0 1 0 516165529 18763776 389 33554432000 4194304 4287625 140734631390336 18446744073709551615 230809185807 0 0 4096 65536 2962849452177489919 0 0 17 1 0 0 0
/proc/1750/statm: 4581 389 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+1.10049 s]
/proc/loadavg: 3.42 5.37 6.83 9/181 1750
/proc/meminfo: memFree=18268164/32951124 swapFree=67111528/67111528
[pid=1750] ppid=1748 vsize=18324 CPUtime=0 cores=1,3,5,7
/proc/1750/stat : 1750 (ppfolio) S 1748 1750 32317 0 -1 4202496 518 0 0 0 0 0 0 0 19 0 1 0 516165529 18763776 389 33554432000 4194304 4287625 140734631390336 18446744073709551615 230809185807 0 0 4096 65536 2962849452177489919 0 0 17 1 0 0 0
/proc/1750/statm: 4581 389 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+1.20047 s]
/proc/loadavg: 3.42 5.37 6.83 9/181 1750
/proc/meminfo: memFree=18268164/32951124 swapFree=67111528/67111528
[pid=1750] ppid=1748 vsize=18324 CPUtime=0 cores=1,3,5,7
/proc/1750/stat : 1750 (ppfolio) S 1748 1750 32317 0 -1 4202496 518 0 0 0 0 0 0 0 19 0 1 0 516165529 18763776 389 33554432000 4194304 4287625 140734631390336 18446744073709551615 230809185807 0 0 4096 65536 2962849452177489919 0 0 17 1 0 0 0
/proc/1750/statm: 4581 389 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+1.50042 s]
/proc/loadavg: 3.42 5.37 6.83 9/181 1750
/proc/meminfo: memFree=18268164/32951124 swapFree=67111528/67111528
[pid=1750] ppid=1748 vsize=18324 CPUtime=0 cores=1,3,5,7
/proc/1750/stat : 1750 (ppfolio) S 1748 1750 32317 0 -1 4202496 518 0 0 0 0 0 0 0 17 0 1 0 516165529 18763776 389 33554432000 4194304 4287625 140734631390336 18446744073709551615 230809185807 0 0 4096 65536 2962849452177489919 0 0 17 1 0 0 0
/proc/1750/statm: 4581 389 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+3.10713 s]
/proc/loadavg: 3.95 5.45 6.84 11/188 1757
/proc/meminfo: memFree=18210328/32951124 swapFree=67111528/67111528
[pid=1750] ppid=1748 vsize=18324 CPUtime=0 cores=1,3,5,7
/proc/1750/stat : 1750 (ppfolio) S 1748 1750 32317 0 -1 4202496 518 0 0 0 0 0 0 0 15 0 1 0 516165529 18763776 389 33554432000 4194304 4287625 140734631390336 18446744073709551615 230809185807 0 0 4096 65536 2962849452177489919 0 0 17 1 0 0 0
/proc/1750/statm: 4581 389 341 23 0 91 0
[pid=1752] ppid=1750 vsize=1365884 CPUtime=3.09 cores=1
/proc/1752/stat : 1752 (cryptominisat) R 1750 1750 32317 0 -1 4202496 8965 0 0 0 308 1 0 0 25 0 1 0 516165529 1398665216 3711 33554432000 4194304 4587354 140735682311088 18446744073709551615 4356816 0 0 4096 2 0 0 0 17 1 0 0 0
/proc/1752/statm: 341471 3711 337 96 0 338033 0
[pid=1753] ppid=1750 vsize=22752 CPUtime=3.09 cores=5
/proc/1753/stat : 1753 (clasp) R 1750 1750 32317 0 -1 4202496 3207 0 0 0 309 0 0 0 25 0 1 0 516165530 23298048 3012 33554432000 4194304 4785740 140736503105408 18446744073709551615 4287028 0 0 4096 16386 0 0 0 17 5 0 0 0
/proc/1753/statm: 5688 3012 340 145 0 2733 0
[pid=1754] ppid=1750 vsize=455032 CPUtime=1.98 cores=7
/proc/1754/stat : 1754 (TNM) R 1750 1750 32317 0 -1 4202496 1150 0 0 0 198 0 0 0 21 0 1 0 516165530 465952768 1047 33554432000 134512640 134534564 4294028816 18446744073709551615 134524508 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/1754/statm: 113758 1047 91 6 0 112424 0
[pid=1755] ppid=1750 vsize=7284 CPUtime=3.1 cores=3
/proc/1755/stat : 1755 (lingeling) R 1750 1750 32317 0 -1 4202496 1731 0 0 0 310 0 0 0 25 0 1 0 516165530 7458816 1015 33554432000 4194304 4352236 140736109794080 18446744073709551615 4328210 0 0 4096 17506 0 0 0 17 3 0 0 0
/proc/1755/statm: 1821 1015 127 39 0 901 0
[pid=1756] ppid=1750 vsize=10932 CPUtime=1.09 cores=7
/proc/1756/stat : 1756 (march_hi) R 1750 1750 32317 0 -1 4202496 4745 0 0 0 109 0 0 0 25 0 1 0 516165530 11194368 1065 33554432000 4194304 4331988 140737486695952 18446744073709551615 4271184 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/1756/statm: 2733 1065 150 34 0 1175 0
Current children cumulated CPU time (s) 12.35
Current children cumulated vsize (KiB) 1880208

[startup+6.30054 s]
/proc/loadavg: 3.95 5.45 6.84 11/188 1758
/proc/meminfo: memFree=18185548/32951124 swapFree=67111528/67111528
[pid=1750] ppid=1748 vsize=18324 CPUtime=0 cores=1,3,5,7
/proc/1750/stat : 1750 (ppfolio) S 1748 1750 32317 0 -1 4202496 518 0 0 0 0 0 0 0 15 0 1 0 516165529 18763776 389 33554432000 4194304 4287625 140734631390336 18446744073709551615 230809185807 0 0 4096 65536 2962849452177489919 0 0 17 1 0 0 0
/proc/1750/statm: 4581 389 341 23 0 91 0
[pid=1752] ppid=1750 vsize=1368924 CPUtime=6.29 cores=1
/proc/1752/stat : 1752 (cryptominisat) R 1750 1750 32317 0 -1 4202496 12679 0 0 0 627 2 0 0 25 0 1 0 516165529 1401778176 6192 33554432000 4194304 4587354 140735682311088 18446744073709551615 4238591 0 0 4096 2 0 0 0 17 1 0 0 0
/proc/1752/statm: 342231 6192 337 96 0 338793 0
[pid=1753] ppid=1750 vsize=24688 CPUtime=6.29 cores=5
/proc/1753/stat : 1753 (clasp) R 1750 1750 32317 0 -1 4202496 3672 0 0 0 629 0 0 0 25 0 1 0 516165530 25280512 3477 33554432000 4194304 4785740 140736503105408 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 5 0 0 0
/proc/1753/statm: 6172 3477 340 145 0 3217 0
[pid=1754] ppid=1750 vsize=455032 CPUtime=3.56 cores=7
/proc/1754/stat : 1754 (TNM) R 1750 1750 32317 0 -1 4202496 1150 0 0 0 356 0 0 0 25 0 1 0 516165530 465952768 1047 33554432000 134512640 134534564 4294028816 18446744073709551615 134524700 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/1754/statm: 113758 1047 91 6 0 112424 0
[pid=1755] ppid=1750 vsize=8504 CPUtime=6.29 cores=3
/proc/1755/stat : 1755 (lingeling) R 1750 1750 32317 0 -1 4202496 2029 0 0 0 629 0 0 0 25 0 1 0 516165530 8708096 1313 33554432000 4194304 4352236 140736109794080 18446744073709551615 4328308 0 0 4096 17506 0 0 0 17 3 0 0 0
/proc/1755/statm: 2126 1313 127 39 0 1206 0
[pid=1756] ppid=1750 vsize=14320 CPUtime=2.7 cores=7
/proc/1756/stat : 1756 (march_hi) R 1750 1750 32317 0 -1 4202496 5787 0 0 0 269 1 0 0 25 0 1 0 516165530 14663680 1732 33554432000 4194304 4331988 140737486695952 18446744073709551615 230808862128 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/1756/statm: 3580 1732 174 34 0 2022 0
Current children cumulated CPU time (s) 25.13
Current children cumulated vsize (KiB) 1889792

[startup+12.7004 s]
/proc/loadavg: 4.43 5.52 6.86 11/188 1758
/proc/meminfo: memFree=18166436/32951124 swapFree=67111528/67111528
[pid=1750] ppid=1748 vsize=18324 CPUtime=0 cores=1,3,5,7
/proc/1750/stat : 1750 (ppfolio) S 1748 1750 32317 0 -1 4202496 518 0 0 0 0 0 0 0 15 0 1 0 516165529 18763776 389 33554432000 4194304 4287625 140734631390336 18446744073709551615 230809185807 0 0 4096 65536 2962849452177489919 0 0 17 1 0 0 0
/proc/1750/statm: 4581 389 341 23 0 91 0
[pid=1752] ppid=1750 vsize=1381208 CPUtime=12.68 cores=1
/proc/1752/stat : 1752 (cryptominisat) R 1750 1750 32317 0 -1 4202496 14878 0 0 0 1266 2 0 0 25 0 1 0 516165529 1414356992 6993 33554432000 4194304 4587354 140735682311088 18446744073709551615 4238538 0 0 4096 2 0 0 0 17 1 0 0 0
/proc/1752/statm: 345302 6993 337 96 0 341864 0
[pid=1753] ppid=1750 vsize=27824 CPUtime=12.69 cores=5
/proc/1753/stat : 1753 (clasp) R 1750 1750 32317 0 -1 4202496 4426 0 0 0 1268 1 0 0 25 0 1 0 516165530 28491776 4231 33554432000 4194304 4785740 140736503105408 18446744073709551615 4553399 0 0 4096 16386 0 0 0 17 5 0 0 0
/proc/1753/statm: 6956 4231 340 145 0 4001 0
[pid=1754] ppid=1750 vsize=459292 CPUtime=6.73 cores=7
/proc/1754/stat : 1754 (TNM) R 1750 1750 32317 0 -1 4202496 1399 0 0 0 673 0 0 0 25 0 1 0 516165530 470315008 1296 33554432000 134512640 134534564 4294028816 18446744073709551615 134528913 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/1754/statm: 114823 1296 95 6 0 113489 0
[pid=1755] ppid=1750 vsize=8572 CPUtime=12.69 cores=3
/proc/1755/stat : 1755 (lingeling) R 1750 1750 32317 0 -1 4202496 2693 0 0 0 1269 0 0 0 25 0 1 0 516165530 8777728 1337 33554432000 4194304 4352236 140736109794080 18446744073709551615 4329205 0 0 4096 17506 0 0 0 17 3 0 0 0
/proc/1755/statm: 2143 1337 127 39 0 1223 0
[pid=1756] ppid=1750 vsize=14320 CPUtime=5.9 cores=7
/proc/1756/stat : 1756 (march_hi) R 1750 1750 32317 0 -1 4202496 5787 0 0 0 589 1 0 0 25 0 1 0 516165530 14663680 1732 33554432000 4194304 4331988 140737486695952 18446744073709551615 4246346 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/1756/statm: 3580 1732 174 34 0 2022 0
Current children cumulated CPU time (s) 50.69
Current children cumulated vsize (KiB) 1909540

[startup+25.501 s]
/proc/loadavg: 5.73 5.76 6.92 11/188 1758
/proc/meminfo: memFree=18160444/32951124 swapFree=67111528/67111528
[pid=1750] ppid=1748 vsize=18324 CPUtime=0 cores=1,3,5,7
/proc/1750/stat : 1750 (ppfolio) S 1748 1750 32317 0 -1 4202496 518 0 0 0 0 0 0 0 15 0 1 0 516165529 18763776 389 33554432000 4194304 4287625 140734631390336 18446744073709551615 230809185807 0 0 4096 65536 2962849452177489919 0 0 17 1 0 0 0
/proc/1750/statm: 4581 389 341 23 0 91 0
[pid=1752] ppid=1750 vsize=1358940 CPUtime=25.49 cores=1
/proc/1752/stat : 1752 (cryptominisat) R 1750 1750 32317 0 -1 4202496 21393 0 0 0 2544 5 0 0 25 0 1 0 516165529 1391554560 3044 33554432000 4194304 4587354 140735682311088 18446744073709551615 230808862406 0 0 4096 2 0 0 0 17 1 0 0 0
/proc/1752/statm: 339735 3044 337 96 0 336297 0
[pid=1753] ppid=1750 vsize=32696 CPUtime=25.49 cores=5
/proc/1753/stat : 1753 (clasp) R 1750 1750 32317 0 -1 4202496 5551 0 0 0 2548 1 0 0 25 0 1 0 516165530 33480704 5356 33554432000 4194304 4785740 140736503105408 18446744073709551615 4286973 0 0 4096 16386 0 0 0 17 5 0 0 0
/proc/1753/statm: 8174 5356 340 145 0 5219 0
[pid=1754] ppid=1750 vsize=459292 CPUtime=13.1 cores=7
/proc/1754/stat : 1754 (TNM) R 1750 1750 32317 0 -1 4202496 1399 0 0 0 1310 0 0 0 25 0 1 0 516165530 470315008 1296 33554432000 134512640 134534564 4294028816 18446744073709551615 134528979 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/1754/statm: 114823 1296 95 6 0 113489 0
[pid=1755] ppid=1750 vsize=10560 CPUtime=25.49 cores=3
/proc/1755/stat : 1755 (lingeling) R 1750 1750 32317 0 -1 4202496 4550 0 0 0 2549 0 0 0 25 0 1 0 516165530 10813440 1834 33554432000 4194304 4352236 140736109794080 18446744073709551615 4328105 0 0 4096 17506 0 0 0 17 3 0 0 0
/proc/1755/statm: 2640 1834 127 39 0 1720 0
[pid=1756] ppid=1750 vsize=14320 CPUtime=12.25 cores=7
/proc/1756/stat : 1756 (march_hi) R 1750 1750 32317 0 -1 4202496 5787 0 0 0 1224 1 0 0 25 0 1 0 516165530 14663680 1732 33554432000 4194304 4331988 140737486695952 18446744073709551615 230808560795 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/1756/statm: 3580 1732 174 34 0 2022 0
Current children cumulated CPU time (s) 101.82
Current children cumulated vsize (KiB) 1894132

[startup+51.1003 s]
/proc/loadavg: 7.26 6.11 7.00 11/188 1759
/proc/meminfo: memFree=18125512/32951124 swapFree=67111528/67111528
[pid=1750] ppid=1748 vsize=18324 CPUtime=0 cores=1,3,5,7
/proc/1750/stat : 1750 (ppfolio) S 1748 1750 32317 0 -1 4202496 518 0 0 0 0 0 0 0 15 0 1 0 516165529 18763776 389 33554432000 4194304 4287625 140734631390336 18446744073709551615 230809185807 0 0 4096 65536 2962849452177489919 0 0 17 1 0 0 0
/proc/1750/statm: 4581 389 341 23 0 91 0
[pid=1752] ppid=1750 vsize=1372000 CPUtime=51.08 cores=1
/proc/1752/stat : 1752 (cryptominisat) R 1750 1750 32317 0 -1 4202496 41739 0 0 0 5099 9 0 0 25 0 1 0 516165529 1404928000 6887 33554432000 4194304 4587354 140735682311088 18446744073709551615 4238467 0 0 4096 2 0 0 0 17 1 0 0 0
/proc/1752/statm: 343000 6887 337 96 0 339562 0
[pid=1753] ppid=1750 vsize=38124 CPUtime=51.09 cores=5
/proc/1753/stat : 1753 (clasp) R 1750 1750 32317 0 -1 4202496 6793 0 0 0 5108 1 0 0 25 0 1 0 516165530 39038976 6598 33554432000 4194304 4785740 140736503105408 18446744073709551615 4550348 0 0 4096 16386 0 0 0 17 5 0 0 0
/proc/1753/statm: 9531 6598 340 145 0 6576 0
[pid=1754] ppid=1750 vsize=459292 CPUtime=25.81 cores=7
/proc/1754/stat : 1754 (TNM) R 1750 1750 32317 0 -1 4202496 1399 0 0 0 2581 0 0 0 25 0 1 0 516165530 470315008 1296 33554432000 134512640 134534564 4294028816 18446744073709551615 134530538 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/1754/statm: 114823 1296 95 6 0 113489 0
[pid=1755] ppid=1750 vsize=9672 CPUtime=51.08 cores=3
/proc/1755/stat : 1755 (lingeling) R 1750 1750 32317 0 -1 4202496 6668 0 0 0 5108 0 0 0 25 0 1 0 516165530 9904128 1612 33554432000 4194304 4352236 140736109794080 18446744073709551615 4329086 0 0 4096 17506 0 0 0 17 3 0 0 0
/proc/1755/statm: 2418 1612 127 39 0 1498 0
[pid=1756] ppid=1750 vsize=16620 CPUtime=24.99 cores=7
/proc/1756/stat : 1756 (march_hi) R 1750 1750 32317 0 -1 4202496 6315 0 0 0 2498 1 0 0 25 0 1 0 516165530 17018880 2260 33554432000 4194304 4331988 140737486695952 18446744073709551615 230808561350 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/1756/statm: 4155 2260 175 34 0 2597 0
Current children cumulated CPU time (s) 204.05
Current children cumulated vsize (KiB) 1914032

[startup+102.301 s]
/proc/loadavg: 8.86 6.73 7.16 11/188 1760
/proc/meminfo: memFree=18092632/32951124 swapFree=67111528/67111528
[pid=1750] ppid=1748 vsize=18324 CPUtime=0 cores=1,3,5,7
/proc/1750/stat : 1750 (ppfolio) S 1748 1750 32317 0 -1 4202496 518 0 0 0 0 0 0 0 15 0 1 0 516165529 18763776 389 33554432000 4194304 4287625 140734631390336 18446744073709551615 230809185807 0 0 4096 65536 2962849452177489919 0 0 17 1 0 0 0
/proc/1750/statm: 4581 389 341 23 0 91 0
[pid=1752] ppid=1750 vsize=1405424 CPUtime=102.29 cores=1

################
# More data... #
################


[startup+1002.3 s]
/proc/loadavg: 8.19 8.20 7.96 9/186 1796
/proc/meminfo: memFree=17970716/32951124 swapFree=67111528/67111528
[pid=1750] ppid=1748 vsize=18324 CPUtime=368.65 cores=1,3,5,7
/proc/1750/stat : 1750 (ppfolio) S 1748 1750 32317 0 -1 4202496 519 183416 0 0 0 0 36806 59 15 0 1 0 516165529 18763776 390 33554432000 4194304 4287625 140734631390336 18446744073709551615 230809185807 0 0 4096 65536 9224195742862639364 0 0 17 1 0 0 0
/proc/1750/statm: 4581 390 342 23 0 91 0
[pid=1753] ppid=1750 vsize=96240 CPUtime=1002.32 cores=5
/proc/1753/stat : 1753 (clasp) R 1750 1750 32317 0 -1 4202496 20294 0 0 0 100227 5 0 0 25 0 1 0 516165530 98549760 19611 33554432000 4194304 4785740 140736503105408 18446744073709551615 4278093 0 0 4096 16386 0 0 0 17 5 0 0 0
/proc/1753/statm: 24060 19611 341 145 0 21105 0
[pid=1754] ppid=1750 vsize=459292 CPUtime=498.83 cores=7
/proc/1754/stat : 1754 (TNM) R 1750 1750 32317 0 -1 4202496 1399 0 0 0 49883 0 0 0 25 0 1 0 516165530 470315008 1296 33554432000 134512640 134534564 4294028816 18446744073709551615 134530598 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/1754/statm: 114823 1296 95 6 0 113489 0
[pid=1755] ppid=1750 vsize=22864 CPUtime=1002.32 cores=3
/proc/1755/stat : 1755 (lingeling) R 1750 1750 32317 0 -1 4202496 39662 0 0 0 100227 5 0 0 25 0 1 0 516165530 23412736 4910 33554432000 4194304 4352236 140736109794080 18446744073709551615 4328390 0 0 4096 17506 0 0 0 17 3 0 0 0
/proc/1755/statm: 5716 4910 127 39 0 4796 0
[pid=1756] ppid=1750 vsize=73256 CPUtime=497.99 cores=7
/proc/1756/stat : 1756 (march_hi) R 1750 1750 32317 0 -1 4202496 18587 0 0 0 49791 8 0 0 25 0 1 0 516165530 75014144 14532 33554432000 4194304 4331988 140737486695952 18446744073709551615 4237419 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/1756/statm: 18314 14532 175 34 0 16756 0
Current children cumulated CPU time (s) 3370.11
Current children cumulated vsize (KiB) 669976

[startup+1062.3 s]
/proc/loadavg: 8.11 8.18 7.96 9/186 1798
/proc/meminfo: memFree=17965608/32951124 swapFree=67111528/67111528
[pid=1750] ppid=1748 vsize=18324 CPUtime=368.65 cores=1,3,5,7
/proc/1750/stat : 1750 (ppfolio) S 1748 1750 32317 0 -1 4202496 519 183416 0 0 0 0 36806 59 15 0 1 0 516165529 18763776 390 33554432000 4194304 4287625 140734631390336 18446744073709551615 230809185807 0 0 4096 65536 9224195742862639364 0 0 17 1 0 0 0
/proc/1750/statm: 4581 390 342 23 0 91 0
[pid=1753] ppid=1750 vsize=97092 CPUtime=1062.32 cores=5
/proc/1753/stat : 1753 (clasp) R 1750 1750 32317 0 -1 4202496 20506 0 0 0 106227 5 0 0 25 0 1 0 516165530 99422208 19823 33554432000 4194304 4785740 140736503105408 18446744073709551615 4287204 0 0 4096 16386 0 0 0 17 5 0 0 0
/proc/1753/statm: 24273 19823 341 145 0 21318 0
[pid=1754] ppid=1750 vsize=459292 CPUtime=528.68 cores=7
/proc/1754/stat : 1754 (TNM) R 1750 1750 32317 0 -1 4202496 1399 0 0 0 52868 0 0 0 25 0 1 0 516165530 470315008 1296 33554432000 134512640 134534564 4294028816 18446744073709551615 134528504 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/1754/statm: 114823 1296 95 6 0 113489 0
[pid=1755] ppid=1750 vsize=18000 CPUtime=1062.32 cores=3
/proc/1755/stat : 1755 (lingeling) R 1750 1750 32317 0 -1 4202496 40270 0 0 0 106227 5 0 0 25 0 1 0 516165530 18432000 3694 33554432000 4194304 4352236 140736109794080 18446744073709551615 4234468 0 0 4096 17506 0 0 0 17 3 0 0 0
/proc/1755/statm: 4500 3694 127 39 0 3580 0
[pid=1756] ppid=1750 vsize=73256 CPUtime=527.81 cores=7
/proc/1756/stat : 1756 (march_hi) R 1750 1750 32317 0 -1 4202496 19358 0 0 0 52772 9 0 0 25 0 1 0 516165530 75014144 15303 33554432000 4194304 4331988 140737486695952 18446744073709551615 4239564 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/1756/statm: 18314 15303 175 34 0 16756 0
Current children cumulated CPU time (s) 3549.78
Current children cumulated vsize (KiB) 665964

[startup+1122.3 s]
/proc/loadavg: 8.16 8.18 7.97 9/186 1799
/proc/meminfo: memFree=17954428/32951124 swapFree=67111528/67111528
[pid=1750] ppid=1748 vsize=18324 CPUtime=368.65 cores=1,3,5,7
/proc/1750/stat : 1750 (ppfolio) S 1748 1750 32317 0 -1 4202496 519 183416 0 0 0 0 36806 59 15 0 1 0 516165529 18763776 390 33554432000 4194304 4287625 140734631390336 18446744073709551615 230809185807 0 0 4096 65536 9224195742862639364 0 0 17 1 0 0 0
/proc/1750/statm: 4581 390 342 23 0 91 0
[pid=1753] ppid=1750 vsize=101376 CPUtime=1122.32 cores=5
/proc/1753/stat : 1753 (clasp) R 1750 1750 32317 0 -1 4202496 21462 0 0 0 112227 5 0 0 25 0 1 0 516165530 103809024 20779 33554432000 4194304 4785740 140736503105408 18446744073709551615 4286912 0 0 4096 16386 0 0 0 17 5 0 0 0
/proc/1753/statm: 25344 20779 341 145 0 22389 0
[pid=1754] ppid=1750 vsize=459292 CPUtime=558.52 cores=7
/proc/1754/stat : 1754 (TNM) R 1750 1750 32317 0 -1 4202496 1399 0 0 0 55852 0 0 0 25 0 1 0 516165530 470315008 1296 33554432000 134512640 134534564 4294028816 18446744073709551615 134528511 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/1754/statm: 114823 1296 95 6 0 113489 0
[pid=1755] ppid=1750 vsize=18000 CPUtime=1122.32 cores=3
/proc/1755/stat : 1755 (lingeling) R 1750 1750 32317 0 -1 4202496 42094 0 0 0 112227 5 0 0 25 0 1 0 516165530 18432000 3694 33554432000 4194304 4352236 140736109794080 18446744073709551615 4328308 0 0 4096 17506 0 0 0 17 3 0 0 0
/proc/1755/statm: 4500 3694 127 39 0 3580 0
[pid=1756] ppid=1750 vsize=73256 CPUtime=557.64 cores=7
/proc/1756/stat : 1756 (march_hi) R 1750 1750 32317 0 -1 4202496 20187 0 0 0 55754 10 0 0 25 0 1 0 516165530 75014144 16132 33554432000 4194304 4331988 140737486695952 18446744073709551615 4265452 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/1756/statm: 18314 16132 175 34 0 16756 0
Current children cumulated CPU time (s) 3729.45
Current children cumulated vsize (KiB) 670248

[startup+1182.31 s]
/proc/loadavg: 8.17 8.17 7.98 9/186 1801
/proc/meminfo: memFree=17942972/32951124 swapFree=67111528/67111528
[pid=1750] ppid=1748 vsize=18324 CPUtime=368.65 cores=1,3,5,7
/proc/1750/stat : 1750 (ppfolio) S 1748 1750 32317 0 -1 4202496 519 183416 0 0 0 0 36806 59 15 0 1 0 516165529 18763776 390 33554432000 4194304 4287625 140734631390336 18446744073709551615 230809185807 0 0 4096 65536 9224195742862639364 0 0 17 1 0 0 0
/proc/1750/statm: 4581 390 342 23 0 91 0
[pid=1753] ppid=1750 vsize=103704 CPUtime=1182.32 cores=5
/proc/1753/stat : 1753 (clasp) R 1750 1750 32317 0 -1 4202496 21961 0 0 0 118227 5 0 0 25 0 1 0 516165530 106192896 21278 33554432000 4194304 4785740 140736503105408 18446744073709551615 4287017 0 0 4096 16386 0 0 0 17 5 0 0 0
/proc/1753/statm: 25926 21278 341 145 0 22971 0
[pid=1754] ppid=1750 vsize=459292 CPUtime=588.34 cores=7
/proc/1754/stat : 1754 (TNM) R 1750 1750 32317 0 -1 4202496 1399 0 0 0 58834 0 0 0 25 0 1 0 516165530 470315008 1296 33554432000 134512640 134534564 4294028816 18446744073709551615 134530144 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/1754/statm: 114823 1296 95 6 0 113489 0
[pid=1755] ppid=1750 vsize=17616 CPUtime=1182.33 cores=3
/proc/1755/stat : 1755 (lingeling) R 1750 1750 32317 0 -1 4202496 43822 0 0 0 118227 6 0 0 25 0 1 0 516165530 18038784 3598 33554432000 4194304 4352236 140736109794080 18446744073709551615 4328085 0 0 4096 17506 0 0 0 17 3 0 0 0
/proc/1755/statm: 4404 3598 127 39 0 3484 0
[pid=1756] ppid=1750 vsize=102724 CPUtime=587.51 cores=7
/proc/1756/stat : 1756 (march_hi) R 1750 1750 32317 0 -1 4202496 20937 0 0 0 58740 11 0 0 25 0 1 0 516165530 105189376 16882 33554432000 4194304 4331988 140737486695952 18446744073709551615 4237416 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/1756/statm: 25681 16882 175 34 0 24123 0
Current children cumulated CPU time (s) 3909.15
Current children cumulated vsize (KiB) 701660

[startup+1242.3 s]
/proc/loadavg: 8.16 8.17 8.00 9/186 1802
/proc/meminfo: memFree=17930436/32951124 swapFree=67111528/67111528
[pid=1750] ppid=1748 vsize=18324 CPUtime=368.65 cores=1,3,5,7
/proc/1750/stat : 1750 (ppfolio) S 1748 1750 32317 0 -1 4202496 519 183416 0 0 0 0 36806 59 15 0 1 0 516165529 18763776 390 33554432000 4194304 4287625 140734631390336 18446744073709551615 230809185807 0 0 4096 65536 9224195742862639364 0 0 17 1 0 0 0
/proc/1750/statm: 4581 390 342 23 0 91 0
[pid=1753] ppid=1750 vsize=106264 CPUtime=1242.32 cores=5
/proc/1753/stat : 1753 (clasp) R 1750 1750 32317 0 -1 4202496 22508 0 0 0 124227 5 0 0 25 0 1 0 516165530 108814336 21825 33554432000 4194304 4785740 140736503105408 18446744073709551615 4553440 0 0 4096 16386 0 0 0 17 5 0 0 0
/proc/1753/statm: 26566 21825 341 145 0 23611 0
[pid=1754] ppid=1750 vsize=459292 CPUtime=618.21 cores=7
/proc/1754/stat : 1754 (TNM) R 1750 1750 32317 0 -1 4202496 1399 0 0 0 61821 0 0 0 25 0 1 0 516165530 470315008 1296 33554432000 134512640 134534564 4294028816 18446744073709551615 134528926 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/1754/statm: 114823 1296 95 6 0 113489 0
[pid=1755] ppid=1750 vsize=15568 CPUtime=1242.33 cores=3
/proc/1755/stat : 1755 (lingeling) R 1750 1750 32317 0 -1 4202496 45134 0 0 0 124227 6 0 0 25 0 1 0 516165530 15941632 3086 33554432000 4194304 4352236 140736109794080 18446744073709551615 4328105 0 0 4096 17506 0 0 0 17 3 0 0 0
/proc/1755/statm: 3892 3086 127 39 0 2972 0
[pid=1756] ppid=1750 vsize=102724 CPUtime=617.31 cores=7
/proc/1756/stat : 1756 (march_hi) R 1750 1750 32317 0 -1 4202496 21770 0 0 0 61719 12 0 0 25 0 1 0 516165530 105189376 17715 33554432000 4194304 4331988 140737486695952 18446744073709551615 4299269 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/1756/statm: 25681 17715 175 34 0 24123 0
Current children cumulated CPU time (s) 4088.82
Current children cumulated vsize (KiB) 702172



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+1300.01 s]
/proc/loadavg: 8.23 8.18 8.01 6/180 1805
/proc/meminfo: memFree=18099096/32951124 swapFree=67111528/67111528
[pid=1750] ppid=1748 vsize=18324 CPUtime=368.65 cores=1,3,5,7
/proc/1750/stat : 1750 (ppfolio) S 1748 1750 32317 0 -1 4202496 519 183416 0 0 0 0 36806 59 15 0 1 0 516165529 18763776 390 33554432000 4194304 4287625 140734631390336 18446744073709551615 230809185807 0 0 4096 65536 9224195742862639364 0 0 17 1 0 0 0
/proc/1750/statm: 4581 390 342 23 0 91 0
[pid=1753] ppid=1750 vsize=108544 CPUtime=1300.03 cores=5
/proc/1753/stat : 1753 (clasp) R 1750 1750 32317 0 -1 4202496 22961 0 0 0 129998 5 0 0 25 0 1 0 516165530 111149056 22278 33554432000 4194304 4785740 140736503105408 18446744073709551615 4287019 0 0 4096 16386 0 0 0 17 5 0 0 0
/proc/1753/statm: 27136 22278 341 145 0 24181 0
[pid=1754] ppid=1750 vsize=459292 CPUtime=646.91 cores=7
/proc/1754/stat : 1754 (TNM) R 1750 1750 32317 0 -1 4202496 1399 0 0 0 64691 0 0 0 25 0 1 0 516165530 470315008 1296 33554432000 134512640 134534564 4294028816 18446744073709551615 134528891 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/1754/statm: 114823 1296 95 6 0 113489 0
[pid=1755] ppid=1750 vsize=22864 CPUtime=1300.04 cores=3
/proc/1755/stat : 1755 (lingeling) R 1750 1750 32317 0 -1 4202496 46958 0 0 0 129998 6 0 0 25 0 1 0 516165530 23412736 4910 33554432000 4194304 4352236 140736109794080 18446744073709551615 4231046 0 0 4096 17506 0 0 0 17 3 0 0 0
/proc/1755/statm: 5716 4910 127 39 0 4796 0
[pid=1756] ppid=1750 vsize=102724 CPUtime=646 cores=7
/proc/1756/stat : 1756 (march_hi) R 1750 1750 32317 0 -1 4202496 22523 0 0 0 64588 12 0 0 25 0 1 0 516165530 105189376 18468 33554432000 4194304 4331988 140737486695952 18446744073709551615 4299299 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/1756/statm: 25681 18468 175 34 0 24123 0
Current children cumulated CPU time (s) 4261.63
Current children cumulated vsize (KiB) 711748

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1300.01 s]
/proc/loadavg: 8.23 8.18 8.01 6/180 1805
/proc/meminfo: memFree=18099096/32951124 swapFree=67111528/67111528
[pid=1750] ppid=1748 vsize=18324 CPUtime=368.65 cores=1,3,5,7
/proc/1750/stat : 1750 (ppfolio) S 1748 1750 32317 0 -1 4202496 519 183416 0 0 0 0 36806 59 15 0 1 0 516165529 18763776 390 33554432000 4194304 4287625 140734631390336 18446744073709551615 230809185807 0 0 4096 65536 9224195742862639364 0 0 17 1 0 0 0
/proc/1750/statm: 4581 390 342 23 0 91 0
[pid=1753] ppid=1750 vsize=108544 CPUtime=1300.03 cores=5
/proc/1753/stat : 1753 (clasp) R 1750 1750 32317 0 -1 4202496 22961 0 0 0 129998 5 0 0 25 0 1 0 516165530 111149056 22278 33554432000 4194304 4785740 140736503105408 18446744073709551615 4287019 0 0 4096 16386 0 0 0 17 5 0 0 0
/proc/1753/statm: 27136 22278 341 145 0 24181 0
[pid=1754] ppid=1750 vsize=459292 CPUtime=646.91 cores=7
/proc/1754/stat : 1754 (TNM) R 1750 1750 32317 0 -1 4202496 1399 0 0 0 64691 0 0 0 25 0 1 0 516165530 470315008 1296 33554432000 134512640 134534564 4294028816 18446744073709551615 134528891 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/1754/statm: 114823 1296 95 6 0 113489 0
[pid=1755] ppid=1750 vsize=22864 CPUtime=1300.04 cores=3
/proc/1755/stat : 1755 (lingeling) R 1750 1750 32317 0 -1 4202496 46958 0 0 0 129998 6 0 0 25 0 1 0 516165530 23412736 4910 33554432000 4194304 4352236 140736109794080 18446744073709551615 4231046 0 0 4096 17506 0 0 0 17 3 0 0 0
/proc/1755/statm: 5716 4910 127 39 0 4796 0
[pid=1756] ppid=1750 vsize=102724 CPUtime=646 cores=7
/proc/1756/stat : 1756 (march_hi) R 1750 1750 32317 0 -1 4202496 22523 0 0 0 64588 12 0 0 25 0 1 0 516165530 105189376 18468 33554432000 4194304 4331988 140737486695952 18446744073709551615 4299299 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/1756/statm: 25681 18468 175 34 0 24123 0
Current children cumulated CPU time (s) 4261.63
Current children cumulated vsize (KiB) 711748

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 1750 and gives
#  childrusage.ru_utime.tv_sec=2960
#  childrusage.ru_utime.tv_usec=843883
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=796878
# CPU time returned by wait4() is 2961.64
# while last known CPU time is 4261.63
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1300.02
CPU time (s): 4261.63
CPU user time (s): 4260.81
CPU system time (s): 0.82
CPU usage (%): 327.813
Max. virtual memory (cumulated for all children) (KiB): 14401052

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 2960.84
system time used= 0.796878
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 254817
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 402
involuntary context switches= 43704

runsolver used 2.18567 second user time and 4.91325 second system time

The end

Launcher Data 

Begin job on node104 at 2011-04-29 04:54:10
IDJOB=3190929
IDBENCH=82660
IDSOLVER=1612
FILE ID=node104/3190929-1304045650
RUNJOBID= node104-1304045650-1727
PBS_JOBID= 13170380
Free space on /tmp= 71544 MiB

SOLVER NAME= ppfolio par
BENCH NAME= SAT11/random/large/unif-k7-r85-v300-c25500-S69399346-065-UNKNOWN.cnf
COMMAND LINE= ppfolio BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3190929-1304045650/watcher-3190929-1304045650 -o /tmp/evaluation-result-3190929-1304045650/solver-3190929-1304045650 -C 4800 -W 1300 -M 15500  ppfolio HOME/instance-3190929-1304045650.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 98486271f2f72889c040eabfd36ebd13
RANDOM SEED=837727944

node104.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.71
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5293.18
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      18286380 kB
Buffers:       2195692 kB
Cached:       11815860 kB
SwapCached:          0 kB
Active:        2351264 kB
Inactive:     11727692 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      18286380 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:           21468 kB
Writeback:           0 kB
AnonPages:       66912 kB
Mapped:          16040 kB
Slab:           523260 kB
PageTables:       5084 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   661876 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71540 MiB
End job on node104 at 2011-04-29 05:15:53