Trace number 3291514

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ppfolio par? (TO) 4800.15 1201.71

General information on the benchmark

NameSAT03/industrial/li/li-test4/
li-test4-100.shuffled-as.sat03-370.cnf
MD5SUM5e5606ae7367b9272c60adc115594912
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables36809
Number of clauses142491
Sum of the clauses size559708
Maximum clause length172
Minimum clause length1
Number of clauses of size 114
Number of clauses of size 224558
Number of clauses of size 367458
Number of clauses of size 449000
Number of clauses of size 51
Number of clauses of size over 51460

Solver Data 

0.00/0.85	This is pico-portfolio, a naive parallel portfolio
0.00/0.85	Copyright (C) 2011 Olivier Roussel
0.00/0.85	Using 4 processing units
0.00/0.85	This portfolio uses the following solvers:
0.00/0.85	[1] cryptominisat 2.7.1 (Mate Soos)
0.00/0.85	[2] clasp 1.3.6 (Martin Gebser, Benjamin Kaufmann, and Torsten Schaub)
0.00/0.85	[3] TNM 2009 (Wanxia Wei and Chu Min Li)
0.00/0.85	[4] lingeling 276-6264d55-100731 (Armin Biere)
0.00/0.85	[5] march_hi 2009 (Marijn Heule and Hans Van Maaren)
0.00/0.85	Starting cryptominisat 2.7.1 on core(s) 0
0.00/0.85	Starting clasp 1.3.6 on core(s) 4
0.00/0.85	Starting TNM 2009 on core(s) 6
0.00/0.85	Starting lingeling 276-6264d55-100731 on core(s) 2
0.00/0.85	Starting march_hi 2009 on core(s) 6

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3291514-1304093023/watcher-3291514-1304093023 -o /tmp/evaluation-result-3291514-1304093023/solver-3291514-1304093023 -C 4800 -W 1300 -M 15500 ppfolio HOME/instance-3291514-1304093023.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 10.28 10.22 9.98 6/182 20857
/proc/meminfo: memFree=18137800/32951124 swapFree=67110396/67111528
[pid=20857] ppid=20855 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/20857/stat : 20857 (ppfolio) R 20855 20857 20607 0 -1 4202496 517 0 0 0 0 0 0 0 25 0 1 0 599533030 18763776 389 33554432000 4194304 4287625 140734978354832 18446744073709551615 224284945935 0 0 4096 65536 0 0 0 17 2 0 0 0
/proc/20857/statm: 4581 389 341 23 0 91 0

[startup+0.236471 s]
/proc/loadavg: 10.28 10.22 9.98 6/182 20857
/proc/meminfo: memFree=18137800/32951124 swapFree=67110396/67111528
[pid=20857] ppid=20855 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/20857/stat : 20857 (ppfolio) S 20855 20857 20607 0 -1 4202496 518 0 0 0 0 0 0 0 25 0 1 0 599533030 18763776 389 33554432000 4194304 4287625 140734978354832 18446744073709551615 224284945935 0 0 4096 65536 3012134648575688703 0 0 17 2 0 0 0
/proc/20857/statm: 4581 389 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.300464 s]
/proc/loadavg: 10.28 10.22 9.98 6/182 20857
/proc/meminfo: memFree=18137800/32951124 swapFree=67110396/67111528
[pid=20857] ppid=20855 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/20857/stat : 20857 (ppfolio) S 20855 20857 20607 0 -1 4202496 518 0 0 0 0 0 0 0 25 0 1 0 599533030 18763776 389 33554432000 4194304 4287625 140734978354832 18446744073709551615 224284945935 0 0 4096 65536 3012134648575688703 0 0 17 2 0 0 0
/proc/20857/statm: 4581 389 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.400445 s]
/proc/loadavg: 10.28 10.22 9.98 6/182 20857
/proc/meminfo: memFree=18137800/32951124 swapFree=67110396/67111528
[pid=20857] ppid=20855 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/20857/stat : 20857 (ppfolio) S 20855 20857 20607 0 -1 4202496 518 0 0 0 0 0 0 0 25 0 1 0 599533030 18763776 389 33554432000 4194304 4287625 140734978354832 18446744073709551615 224284945935 0 0 4096 65536 3012134648575688703 0 0 17 2 0 0 0
/proc/20857/statm: 4581 389 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.700398 s]
/proc/loadavg: 10.28 10.22 9.98 6/182 20857
/proc/meminfo: memFree=18137800/32951124 swapFree=67110396/67111528
[pid=20857] ppid=20855 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/20857/stat : 20857 (ppfolio) S 20855 20857 20607 0 -1 4202496 518 0 0 0 0 0 0 0 19 0 1 0 599533030 18763776 389 33554432000 4194304 4287625 140734978354832 18446744073709551615 224284945935 0 0 4096 65536 3012134648575688703 0 0 17 2 0 0 0
/proc/20857/statm: 4581 389 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+1.5013 s]
/proc/loadavg: 10.28 10.22 9.98 6/181 20864
/proc/meminfo: memFree=18273836/32951124 swapFree=67110396/67111528
[pid=20857] ppid=20855 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/20857/stat : 20857 (ppfolio) S 20855 20857 20607 0 -1 4202496 518 0 0 0 0 0 0 0 15 0 1 0 599533030 18763776 389 33554432000 4194304 4287625 140734978354832 18446744073709551615 224284945935 0 0 4096 65536 3012134648575688703 0 0 17 2 0 0 0
/proc/20857/statm: 4581 389 341 23 0 91 0
[pid=20858] ppid=20857 vsize=58284 CPUtime=1.49 cores=0
/proc/20858/stat : 20858 (cryptominisat) R 20857 20857 20607 0 -1 4202496 13306 0 0 0 147 2 0 0 25 0 1 0 599533030 59682816 9783 33554432000 4194304 4587354 140737270376096 18446744073709551615 4237679 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/20858/statm: 14571 9783 330 96 0 11133 0
[pid=20859] ppid=20857 vsize=38240 CPUtime=1.49 cores=4
/proc/20859/stat : 20859 (clasp) R 20857 20857 20607 0 -1 4202496 8675 0 0 0 148 1 0 0 20 0 1 0 599533030 39157760 6726 33554432000 4194304 4785740 140733566989952 18446744073709551615 4553402 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/20859/statm: 9560 6726 339 145 0 6605 0
[pid=20860] ppid=20857 vsize=463744 CPUtime=0.72 cores=6
/proc/20860/stat : 20860 (TNM) R 20857 20857 20607 0 -1 4202496 3959 0 0 0 72 0 0 0 25 0 1 0 599533030 474873856 3856 33554432000 134512640 134534564 4287654976 18446744073709551615 134524546 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/20860/statm: 115936 3856 91 6 0 114602 0
[pid=20861] ppid=20857 vsize=21464 CPUtime=1.49 cores=2
/proc/20861/stat : 20861 (lingeling) R 20857 20857 20607 0 -1 4202496 6579 0 0 0 148 1 0 0 25 0 1 0 599533030 21979136 4530 33554432000 4194304 4352236 140735089990432 18446744073709551615 4250383 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/20861/statm: 5366 4530 124 39 0 4446 0
[pid=20862] ppid=20857 vsize=34476 CPUtime=0.75 cores=6
/proc/20862/stat : 20862 (march_hi) R 20857 20857 20607 0 -1 4194304 16424 0 0 0 74 1 0 0 25 0 1 0 599533030 35303424 5806 33554432000 4194304 4331988 140736594304160 18446744073709551615 4252197 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/20862/statm: 8619 5806 129 34 0 7061 0
Current children cumulated CPU time (s) 5.94
Current children cumulated vsize (KiB) 634532

[startup+3.10105 s]
/proc/loadavg: 9.85 10.14 9.95 6/181 20864
/proc/meminfo: memFree=18278176/32951124 swapFree=67110396/67111528
[pid=20857] ppid=20855 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/20857/stat : 20857 (ppfolio) S 20855 20857 20607 0 -1 4202496 518 0 0 0 0 0 0 0 15 0 1 0 599533030 18763776 389 33554432000 4194304 4287625 140734978354832 18446744073709551615 224284945935 0 0 4096 65536 3012134648575688703 0 0 17 2 0 0 0
/proc/20857/statm: 4581 389 341 23 0 91 0
[pid=20858] ppid=20857 vsize=58284 CPUtime=3.09 cores=0
/proc/20858/stat : 20858 (cryptominisat) R 20857 20857 20607 0 -1 4202496 13343 0 0 0 307 2 0 0 25 0 1 0 599533030 59682816 9815 33554432000 4194304 4587354 140737270376096 18446744073709551615 4241427 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/20858/statm: 14571 9815 331 96 0 11133 0
[pid=20859] ppid=20857 vsize=38240 CPUtime=3.09 cores=4
/proc/20859/stat : 20859 (clasp) R 20857 20857 20607 0 -1 4202496 8680 0 0 0 308 1 0 0 25 0 1 0 599533030 39157760 6731 33554432000 4194304 4785740 140733566989952 18446744073709551615 4416852 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/20859/statm: 9560 6731 340 145 0 6605 0
[pid=20860] ppid=20857 vsize=463744 CPUtime=1.52 cores=6
/proc/20860/stat : 20860 (TNM) R 20857 20857 20607 0 -1 4202496 3959 0 0 0 152 0 0 0 25 0 1 0 599533030 474873856 3856 33554432000 134512640 134534564 4287654976 18446744073709551615 134524561 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/20860/statm: 115936 3856 91 6 0 114602 0
[pid=20861] ppid=20857 vsize=21464 CPUtime=3.09 cores=2
/proc/20861/stat : 20861 (lingeling) R 20857 20857 20607 0 -1 4202496 6581 0 0 0 308 1 0 0 25 0 1 0 599533030 21979136 4532 33554432000 4194304 4352236 140735089990432 18446744073709551615 4300236 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/20861/statm: 5366 4532 125 39 0 4446 0
[pid=20862] ppid=20857 vsize=33572 CPUtime=1.55 cores=6
/proc/20862/stat : 20862 (march_hi) R 20857 20857 20607 0 -1 4202496 27901 0 0 0 152 3 0 0 25 0 1 0 599533030 34377728 5621 33554432000 4194304 4331988 140736594304160 18446744073709551615 4217714 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/20862/statm: 8393 5621 160 34 0 6835 0
Current children cumulated CPU time (s) 12.34
Current children cumulated vsize (KiB) 633628

[startup+6.30046 s]
/proc/loadavg: 9.85 10.14 9.95 11/188 20896
/proc/meminfo: memFree=18243524/32951124 swapFree=67110396/67111528
[pid=20857] ppid=20855 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/20857/stat : 20857 (ppfolio) S 20855 20857 20607 0 -1 4202496 518 0 0 0 0 0 0 0 15 0 1 0 599533030 18763776 389 33554432000 4194304 4287625 140734978354832 18446744073709551615 224284945935 0 0 4096 65536 3012134648575688703 0 0 17 2 0 0 0
/proc/20857/statm: 4581 389 341 23 0 91 0
[pid=20858] ppid=20857 vsize=58284 CPUtime=6.29 cores=0
/proc/20858/stat : 20858 (cryptominisat) R 20857 20857 20607 0 -1 4202496 13394 0 0 0 627 2 0 0 25 0 1 0 599533030 59682816 9859 33554432000 4194304 4587354 140737270376096 18446744073709551615 4230260 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/20858/statm: 14571 9859 331 96 0 11133 0
[pid=20859] ppid=20857 vsize=38240 CPUtime=6.29 cores=4
/proc/20859/stat : 20859 (clasp) R 20857 20857 20607 0 -1 4202496 8683 0 0 0 628 1 0 0 25 0 1 0 599533030 39157760 6734 33554432000 4194304 4785740 140733566989952 18446744073709551615 4320808 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/20859/statm: 9560 6734 340 145 0 6605 0
[pid=20860] ppid=20857 vsize=463744 CPUtime=3.12 cores=6
/proc/20860/stat : 20860 (TNM) R 20857 20857 20607 0 -1 4202496 3959 0 0 0 312 0 0 0 25 0 1 0 599533030 474873856 3856 33554432000 134512640 134534564 4287654976 18446744073709551615 134524664 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/20860/statm: 115936 3856 91 6 0 114602 0
[pid=20861] ppid=20857 vsize=21464 CPUtime=6.28 cores=2
/proc/20861/stat : 20861 (lingeling) R 20857 20857 20607 0 -1 4202496 6581 0 0 0 627 1 0 0 25 0 1 0 599533030 21979136 4532 33554432000 4194304 4352236 140735089990432 18446744073709551615 4207615 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/20861/statm: 5366 4532 125 39 0 4446 0
[pid=20862] ppid=20857 vsize=37192 CPUtime=3.13 cores=6
/proc/20862/stat : 20862 (march_hi) R 20857 20857 20607 0 -1 4202496 28680 0 0 0 310 3 0 0 25 0 1 0 599533030 38084608 6398 33554432000 4194304 4331988 140736594304160 18446744073709551615 4217696 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/20862/statm: 9298 6398 161 34 0 7740 0
Current children cumulated CPU time (s) 25.11
Current children cumulated vsize (KiB) 637248

[startup+12.7004 s]
/proc/loadavg: 9.95 10.15 9.96 11/188 20896
/proc/meminfo: memFree=18199660/32951124 swapFree=67110396/67111528
[pid=20857] ppid=20855 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/20857/stat : 20857 (ppfolio) S 20855 20857 20607 0 -1 4202496 518 0 0 0 0 0 0 0 15 0 1 0 599533030 18763776 389 33554432000 4194304 4287625 140734978354832 18446744073709551615 224284945935 0 0 4096 65536 3012134648575688703 0 0 17 2 0 0 0
/proc/20857/statm: 4581 389 341 23 0 91 0
[pid=20858] ppid=20857 vsize=58284 CPUtime=12.69 cores=0
/proc/20858/stat : 20858 (cryptominisat) R 20857 20857 20607 0 -1 4202496 13455 0 0 0 1267 2 0 0 25 0 1 0 599533030 59682816 9910 33554432000 4194304 4587354 140737270376096 18446744073709551615 4237740 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/20858/statm: 14571 9910 331 96 0 11133 0
[pid=20859] ppid=20857 vsize=38240 CPUtime=12.69 cores=4
/proc/20859/stat : 20859 (clasp) R 20857 20857 20607 0 -1 4202496 8686 0 0 0 1268 1 0 0 25 0 1 0 599533030 39157760 6737 33554432000 4194304 4785740 140733566989952 18446744073709551615 4552550 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/20859/statm: 9560 6737 340 145 0 6605 0
[pid=20860] ppid=20857 vsize=477232 CPUtime=6.32 cores=6
/proc/20860/stat : 20860 (TNM) R 20857 20857 20607 0 -1 4202496 8771 0 0 0 631 1 0 0 25 0 1 0 599533030 488685568 8668 33554432000 134512640 134534564 4287654976 18446744073709551615 134528726 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/20860/statm: 119308 8668 95 6 0 117974 0
[pid=20861] ppid=20857 vsize=21448 CPUtime=12.68 cores=2
/proc/20861/stat : 20861 (lingeling) R 20857 20857 20607 0 -1 4202496 6583 0 0 0 1267 1 0 0 25 0 1 0 599533030 21962752 4530 33554432000 4194304 4352236 140735089990432 18446744073709551615 4327848 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/20861/statm: 5362 4530 127 39 0 4442 0
[pid=20862] ppid=20857 vsize=41924 CPUtime=6.3 cores=6
/proc/20862/stat : 20862 (march_hi) R 20857 20857 20607 0 -1 4202496 30730 0 0 0 626 4 0 0 25 0 1 0 599533030 42930176 7438 33554432000 4194304 4331988 140736594304160 18446744073709551615 4217714 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/20862/statm: 10481 7438 166 34 0 8923 0
Current children cumulated CPU time (s) 50.68
Current children cumulated vsize (KiB) 655452

[startup+25.5073 s]
/proc/loadavg: 9.96 10.14 9.96 11/188 20896
/proc/meminfo: memFree=18171412/32951124 swapFree=67110396/67111528
[pid=20857] ppid=20855 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/20857/stat : 20857 (ppfolio) S 20855 20857 20607 0 -1 4202496 519 0 0 0 0 0 0 0 15 0 1 0 599533030 18763776 389 33554432000 4194304 4287625 140734978354832 18446744073709551615 224284945935 0 0 4096 65536 3012134648575688703 0 0 17 2 0 0 0
/proc/20857/statm: 4581 389 341 23 0 91 0
[pid=20858] ppid=20857 vsize=61628 CPUtime=25.49 cores=0
/proc/20858/stat : 20858 (cryptominisat) R 20857 20857 20607 0 -1 4202496 15508 0 0 0 2547 2 0 0 25 0 1 0 599533030 63107072 10334 33554432000 4194304 4587354 140737270376096 18446744073709551615 4237467 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/20858/statm: 15407 10334 334 96 0 11969 0
[pid=20859] ppid=20857 vsize=38240 CPUtime=25.5 cores=4
/proc/20859/stat : 20859 (clasp) R 20857 20857 20607 0 -1 4202496 8688 0 0 0 2549 1 0 0 25 0 1 0 599533030 39157760 6739 33554432000 4194304 4785740 140733566989952 18446744073709551615 4553075 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/20859/statm: 9560 6739 340 145 0 6605 0
[pid=20860] ppid=20857 vsize=477232 CPUtime=12.65 cores=6
/proc/20860/stat : 20860 (TNM) R 20857 20857 20607 0 -1 4202496 8771 0 0 0 1264 1 0 0 25 0 1 0 599533030 488685568 8668 33554432000 134512640 134534564 4287654976 18446744073709551615 134530158 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/20860/statm: 119308 8668 95 6 0 117974 0
[pid=20861] ppid=20857 vsize=21448 CPUtime=25.49 cores=2
/proc/20861/stat : 20861 (lingeling) R 20857 20857 20607 0 -1 4202496 6583 0 0 0 2548 1 0 0 25 0 1 0 599533030 21962752 4530 33554432000 4194304 4352236 140735089990432 18446744073709551615 4328009 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/20861/statm: 5362 4530 127 39 0 4442 0
[pid=20862] ppid=20857 vsize=90460 CPUtime=12.69 cores=6
/proc/20862/stat : 20862 (march_hi) R 20857 20857 20607 0 -1 4202496 40769 0 0 0 1264 5 0 0 25 0 1 0 599533030 92631040 12271 33554432000 4194304 4331988 140736594304160 18446744073709551615 4228459 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/20862/statm: 22615 12271 187 34 0 21057 0
Current children cumulated CPU time (s) 101.82
Current children cumulated vsize (KiB) 707332

[startup+51.101 s]
/proc/loadavg: 9.97 10.13 9.96 11/188 20898
/proc/meminfo: memFree=18122272/32951124 swapFree=67110396/67111528
[pid=20857] ppid=20855 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/20857/stat : 20857 (ppfolio) S 20855 20857 20607 0 -1 4202496 519 0 0 0 0 0 0 0 15 0 1 0 599533030 18763776 389 33554432000 4194304 4287625 140734978354832 18446744073709551615 224284945935 0 0 4096 65536 3012134648575688703 0 0 17 2 0 0 0
/proc/20857/statm: 4581 389 341 23 0 91 0
[pid=20858] ppid=20857 vsize=1454644 CPUtime=51.08 cores=0
/proc/20858/stat : 20858 (cryptominisat) R 20857 20857 20607 0 -1 4202496 40126 0 0 0 5099 9 0 0 25 0 1 0 599533030 1489555456 13569 33554432000 4194304 4587354 140737270376096 18446744073709551615 4236402 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/20858/statm: 363661 13569 337 96 0 360223 0
[pid=20859] ppid=20857 vsize=40484 CPUtime=51.08 cores=4
/proc/20859/stat : 20859 (clasp) R 20857 20857 20607 0 -1 4202496 9303 0 0 0 5107 1 0 0 25 0 1 0 599533030 41455616 7354 33554432000 4194304 4785740 140733566989952 18446744073709551615 4552550 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/20859/statm: 10121 7354 340 145 0 7166 0
[pid=20860] ppid=20857 vsize=477232 CPUtime=25.41 cores=6
/proc/20860/stat : 20860 (TNM) R 20857 20857 20607 0 -1 4202496 8771 0 0 0 2540 1 0 0 25 0 1 0 599533030 488685568 8668 33554432000 134512640 134534564 4287654976 18446744073709551615 134528726 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/20860/statm: 119308 8668 95 6 0 117974 0
[pid=20861] ppid=20857 vsize=21988 CPUtime=51.08 cores=2

################
# More data... #
################

/proc/20862/statm: 24579 13837 216 34 0 23021 0
Current children cumulated CPU time (s) 3763.96
Current children cumulated vsize (KiB) 2712520

[startup+1002.31 s]
/proc/loadavg: 10.18 10.18 10.09 11/188 20922
/proc/meminfo: memFree=17246020/32951124 swapFree=67110396/67111528
[pid=20857] ppid=20855 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/20857/stat : 20857 (ppfolio) S 20855 20857 20607 0 -1 4202496 519 0 0 0 0 0 0 0 15 0 1 0 599533030 18763776 389 33554432000 4194304 4287625 140734978354832 18446744073709551615 224284945935 0 0 4096 65536 3012134648575688703 0 0 17 2 0 0 0
/proc/20857/statm: 4581 389 341 23 0 91 0
[pid=20858] ppid=20857 vsize=1663748 CPUtime=1002.31 cores=0
/proc/20858/stat : 20858 (cryptominisat) R 20857 20857 20607 0 -1 4202496 861669 0 0 0 99969 262 0 0 25 0 1 0 599533030 1703677952 83341 33554432000 4194304 4587354 140737270376096 18446744073709551615 4238576 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/20858/statm: 415937 83341 337 96 0 412499 0
[pid=20859] ppid=20857 vsize=491172 CPUtime=1002.21 cores=4
/proc/20859/stat : 20859 (clasp) R 20857 20857 20607 0 -1 4202496 121458 0 0 0 100194 27 0 0 25 0 1 0 599533030 502960128 119509 33554432000 4194304 4785740 140733566989952 18446744073709551615 4278587 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/20859/statm: 122793 119509 340 145 0 119838 0
[pid=20860] ppid=20857 vsize=477232 CPUtime=498.41 cores=6
/proc/20860/stat : 20860 (TNM) R 20857 20857 20607 0 -1 4202496 8771 0 0 0 49840 1 0 0 25 0 1 0 599533030 488685568 8668 33554432000 134512640 134534564 4287654976 18446744073709551615 134530484 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/20860/statm: 119308 8668 95 6 0 117974 0
[pid=20861] ppid=20857 vsize=42224 CPUtime=1002.3 cores=2
/proc/20861/stat : 20861 (lingeling) R 20857 20857 20607 0 -1 4202496 29404 0 0 0 100226 4 0 0 25 0 1 0 599533030 43237376 9724 33554432000 4194304 4352236 140735089990432 18446744073709551615 4328102 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/20861/statm: 10556 9724 127 39 0 9636 0
[pid=20862] ppid=20857 vsize=98316 CPUtime=498.42 cores=6
/proc/20862/stat : 20862 (march_hi) R 20857 20857 20607 0 -1 4202496 42396 0 0 0 49836 6 0 0 25 0 1 0 599533030 100675584 13898 33554432000 4194304 4331988 140736594304160 18446744073709551615 4263139 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/20862/statm: 24579 13898 216 34 0 23021 0
Current children cumulated CPU time (s) 4003.65
Current children cumulated vsize (KiB) 2791016

[startup+1062.3 s]
/proc/loadavg: 10.21 10.19 10.10 11/188 20924
/proc/meminfo: memFree=17274864/32951124 swapFree=67110396/67111528
[pid=20857] ppid=20855 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/20857/stat : 20857 (ppfolio) S 20855 20857 20607 0 -1 4202496 519 0 0 0 0 0 0 0 15 0 1 0 599533030 18763776 389 33554432000 4194304 4287625 140734978354832 18446744073709551615 224284945935 0 0 4096 65536 3012134648575688703 0 0 17 2 0 0 0
/proc/20857/statm: 4581 389 341 23 0 91 0
[pid=20858] ppid=20857 vsize=1677184 CPUtime=1062.3 cores=0
/proc/20858/stat : 20858 (cryptominisat) R 20857 20857 20607 0 -1 4202496 896383 0 0 0 105953 277 0 0 25 0 1 0 599533030 1717436416 68286 33554432000 4194304 4587354 140737270376096 18446744073709551615 4237928 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/20858/statm: 419296 68286 337 96 0 415858 0
[pid=20859] ppid=20857 vsize=522188 CPUtime=1062.2 cores=4
/proc/20859/stat : 20859 (clasp) R 20857 20857 20607 0 -1 4202496 129141 0 0 0 106191 29 0 0 25 0 1 0 599533030 534720512 127192 33554432000 4194304 4785740 140733566989952 18446744073709551615 4553411 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/20859/statm: 130547 127192 340 145 0 127592 0
[pid=20860] ppid=20857 vsize=477232 CPUtime=528.24 cores=6
/proc/20860/stat : 20860 (TNM) R 20857 20857 20607 0 -1 4202496 8771 0 0 0 52823 1 0 0 25 0 1 0 599533030 488685568 8668 33554432000 134512640 134534564 4287654976 18446744073709551615 134528710 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/20860/statm: 119308 8668 95 6 0 117974 0
[pid=20861] ppid=20857 vsize=49408 CPUtime=1062.29 cores=2
/proc/20861/stat : 20861 (lingeling) R 20857 20857 20607 0 -1 4202496 31200 0 0 0 106225 4 0 0 25 0 1 0 599533030 50593792 11520 33554432000 4194304 4352236 140735089990432 18446744073709551615 4328063 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/20861/statm: 12352 11520 127 39 0 11432 0
[pid=20862] ppid=20857 vsize=98316 CPUtime=528.27 cores=6
/proc/20862/stat : 20862 (march_hi) R 20857 20857 20607 0 -1 4202496 42457 0 0 0 52821 6 0 0 25 0 1 0 599533030 100675584 13959 33554432000 4194304 4331988 140736594304160 18446744073709551615 4228459 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/20862/statm: 24579 13959 216 34 0 23021 0
Current children cumulated CPU time (s) 4243.3
Current children cumulated vsize (KiB) 2842652

[startup+1122.3 s]
/proc/loadavg: 10.31 10.21 10.11 11/188 20926
/proc/meminfo: memFree=17282504/32951124 swapFree=67110396/67111528
[pid=20857] ppid=20855 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/20857/stat : 20857 (ppfolio) S 20855 20857 20607 0 -1 4202496 519 0 0 0 0 0 0 0 15 0 1 0 599533030 18763776 389 33554432000 4194304 4287625 140734978354832 18446744073709551615 224284945935 0 0 4096 65536 3012134648575688703 0 0 17 2 0 0 0
/proc/20857/statm: 4581 389 341 23 0 91 0
[pid=20858] ppid=20857 vsize=1622528 CPUtime=1122.31 cores=0
/proc/20858/stat : 20858 (cryptominisat) R 20857 20857 20607 0 -1 4202496 933528 0 0 0 111939 292 0 0 25 0 1 0 599533030 1661468672 60066 33554432000 4194304 4587354 140737270376096 18446744073709551615 4244609 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/20858/statm: 405632 60066 337 96 0 402194 0
[pid=20859] ppid=20857 vsize=528028 CPUtime=1122.19 cores=4
/proc/20859/stat : 20859 (clasp) R 20857 20857 20607 0 -1 4202496 130627 0 0 0 112190 29 0 0 25 0 1 0 599533030 540700672 128678 33554432000 4194304 4785740 140733566989952 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/20859/statm: 132007 128678 340 145 0 129052 0
[pid=20860] ppid=20857 vsize=477232 CPUtime=558.06 cores=6
/proc/20860/stat : 20860 (TNM) R 20857 20857 20607 0 -1 4202496 8771 0 0 0 55805 1 0 0 25 0 1 0 599533030 488685568 8668 33554432000 134512640 134534564 4287654976 18446744073709551615 134525873 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/20860/statm: 119308 8668 95 6 0 117974 0
[pid=20861] ppid=20857 vsize=57616 CPUtime=1122.3 cores=2
/proc/20861/stat : 20861 (lingeling) R 20857 20857 20607 0 -1 4202496 39908 0 0 0 112225 5 0 0 25 0 1 0 599533030 58998784 13572 33554432000 4194304 4352236 140735089990432 18446744073709551615 4328181 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/20861/statm: 14404 13572 127 39 0 13484 0
[pid=20862] ppid=20857 vsize=98316 CPUtime=558.12 cores=6
/proc/20862/stat : 20862 (march_hi) R 20857 20857 20607 0 -1 4202496 42517 0 0 0 55806 6 0 0 25 0 1 0 599533030 100675584 14019 33554432000 4194304 4331988 140736594304160 18446744073709551615 4228538 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/20862/statm: 24579 14019 216 34 0 23021 0
Current children cumulated CPU time (s) 4482.98
Current children cumulated vsize (KiB) 2802044

[startup+1182.3 s]
/proc/loadavg: 10.26 10.21 10.11 11/188 20928
/proc/meminfo: memFree=17220952/32951124 swapFree=67110396/67111528
[pid=20857] ppid=20855 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/20857/stat : 20857 (ppfolio) S 20855 20857 20607 0 -1 4202496 519 0 0 0 0 0 0 0 15 0 1 0 599533030 18763776 389 33554432000 4194304 4287625 140734978354832 18446744073709551615 224284945935 0 0 4096 65536 3012134648575688703 0 0 17 2 0 0 0
/proc/20857/statm: 4581 389 341 23 0 91 0
[pid=20858] ppid=20857 vsize=501544 CPUtime=1182.31 cores=0
/proc/20858/stat : 20858 (cryptominisat) R 20857 20857 20607 0 -1 4202496 984658 0 0 0 117919 312 0 0 25 0 1 0 599533030 513581056 75264 33554432000 4194304 4587354 140737270376096 18446744073709551615 4238593 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/20858/statm: 125386 75264 337 96 0 121948 0
[pid=20859] ppid=20857 vsize=533908 CPUtime=1182.18 cores=4
/proc/20859/stat : 20859 (clasp) R 20857 20857 20607 0 -1 4202496 132127 0 0 0 118189 29 0 0 25 0 1 0 599533030 546721792 130178 33554432000 4194304 4785740 140733566989952 18446744073709551615 4552550 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/20859/statm: 133477 130178 340 145 0 130522 0
[pid=20860] ppid=20857 vsize=477232 CPUtime=587.93 cores=6
/proc/20860/stat : 20860 (TNM) R 20857 20857 20607 0 -1 4202496 8771 0 0 0 58792 1 0 0 25 0 1 0 599533030 488685568 8668 33554432000 134512640 134534564 4287654976 18446744073709551615 134528659 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/20860/statm: 119308 8668 95 6 0 117974 0
[pid=20861] ppid=20857 vsize=56592 CPUtime=1182.29 cores=2
/proc/20861/stat : 20861 (lingeling) R 20857 20857 20607 0 -1 4202496 41956 0 0 0 118224 5 0 0 25 0 1 0 599533030 57950208 13316 33554432000 4194304 4352236 140735089990432 18446744073709551615 4276518 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/20861/statm: 14148 13316 127 39 0 13228 0
[pid=20862] ppid=20857 vsize=98316 CPUtime=587.93 cores=6
/proc/20862/stat : 20862 (march_hi) R 20857 20857 20607 0 -1 4202496 42578 0 0 0 58787 6 0 0 25 0 1 0 599533030 100675584 14080 33554432000 4194304 4331988 140736594304160 18446744073709551615 4263271 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/20862/statm: 24579 14080 216 34 0 23021 0
Current children cumulated CPU time (s) 4722.64
Current children cumulated vsize (KiB) 1685916



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1201.7 s]
/proc/loadavg: 10.23 10.20 10.11 11/188 20928
/proc/meminfo: memFree=17185104/32951124 swapFree=67110396/67111528
[pid=20857] ppid=20855 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/20857/stat : 20857 (ppfolio) S 20855 20857 20607 0 -1 4202496 519 0 0 0 0 0 0 0 15 0 1 0 599533030 18763776 389 33554432000 4194304 4287625 140734978354832 18446744073709551615 224284945935 0 0 4096 65536 3012134648575688703 0 0 17 2 0 0 0
/proc/20857/statm: 4581 389 341 23 0 91 0
[pid=20858] ppid=20857 vsize=501544 CPUtime=1201.71 cores=0
/proc/20858/stat : 20858 (cryptominisat) R 20857 20857 20607 0 -1 4202496 996890 0 0 0 119854 317 0 0 25 0 1 0 599533030 513581056 87474 33554432000 4194304 4587354 140737270376096 18446744073709551615 4241101 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/20858/statm: 125386 87475 337 96 0 121948 0
[pid=20859] ppid=20857 vsize=533908 CPUtime=1201.59 cores=4
/proc/20859/stat : 20859 (clasp) R 20857 20857 20607 0 -1 4202496 132127 0 0 0 120130 29 0 0 25 0 1 0 599533030 546721792 130178 33554432000 4194304 4785740 140733566989952 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/20859/statm: 133477 130178 340 145 0 130522 0
[pid=20860] ppid=20857 vsize=477232 CPUtime=597.54 cores=6
/proc/20860/stat : 20860 (TNM) R 20857 20857 20607 0 -1 4202496 8771 0 0 0 59753 1 0 0 25 0 1 0 599533030 488685568 8668 33554432000 134512640 134534564 4287654976 18446744073709551615 134528710 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/20860/statm: 119308 8668 95 6 0 117974 0
[pid=20861] ppid=20857 vsize=56592 CPUtime=1201.7 cores=2
/proc/20861/stat : 20861 (lingeling) R 20857 20857 20607 0 -1 4202496 41956 0 0 0 120165 5 0 0 25 0 1 0 599533030 57950208 13316 33554432000 4194304 4352236 140735089990432 18446744073709551615 4328022 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/20861/statm: 14148 13316 127 39 0 13228 0
[pid=20862] ppid=20857 vsize=98316 CPUtime=597.61 cores=6
/proc/20862/stat : 20862 (march_hi) R 20857 20857 20607 0 -1 4202496 42598 0 0 0 59755 6 0 0 25 0 1 0 599533030 100675584 14100 33554432000 4194304 4331988 140736594304160 18446744073709551615 4228459 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/20862/statm: 24579 14100 216 34 0 23021 0
Current children cumulated CPU time (s) 4800.15
Current children cumulated vsize (KiB) 1685916

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1201.7 s]
/proc/loadavg: 10.23 10.20 10.11 11/188 20928
/proc/meminfo: memFree=17185104/32951124 swapFree=67110396/67111528
[pid=20857] ppid=20855 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/20857/stat : 20857 (ppfolio) S 20855 20857 20607 0 -1 4202496 519 0 0 0 0 0 0 0 15 0 1 0 599533030 18763776 389 33554432000 4194304 4287625 140734978354832 18446744073709551615 224284945935 0 0 4096 65536 3012134648575688703 0 0 17 2 0 0 0
/proc/20857/statm: 4581 389 341 23 0 91 0
[pid=20858] ppid=20857 vsize=501544 CPUtime=1201.71 cores=0
/proc/20858/stat : 20858 (cryptominisat) R 20857 20857 20607 0 -1 4202496 996890 0 0 0 119854 317 0 0 25 0 1 0 599533030 513581056 87474 33554432000 4194304 4587354 140737270376096 18446744073709551615 4241101 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/20858/statm: 125386 87475 337 96 0 121948 0
[pid=20859] ppid=20857 vsize=533908 CPUtime=1201.59 cores=4
/proc/20859/stat : 20859 (clasp) R 20857 20857 20607 0 -1 4202496 132127 0 0 0 120130 29 0 0 25 0 1 0 599533030 546721792 130178 33554432000 4194304 4785740 140733566989952 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/20859/statm: 133477 130178 340 145 0 130522 0
[pid=20860] ppid=20857 vsize=477232 CPUtime=597.54 cores=6
/proc/20860/stat : 20860 (TNM) R 20857 20857 20607 0 -1 4202496 8771 0 0 0 59753 1 0 0 25 0 1 0 599533030 488685568 8668 33554432000 134512640 134534564 4287654976 18446744073709551615 134528710 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/20860/statm: 119308 8668 95 6 0 117974 0
[pid=20861] ppid=20857 vsize=56592 CPUtime=1201.7 cores=2
/proc/20861/stat : 20861 (lingeling) R 20857 20857 20607 0 -1 4202496 41956 0 0 0 120165 5 0 0 25 0 1 0 599533030 57950208 13316 33554432000 4194304 4352236 140735089990432 18446744073709551615 4328022 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/20861/statm: 14148 13316 127 39 0 13228 0
[pid=20862] ppid=20857 vsize=98316 CPUtime=597.61 cores=6
/proc/20862/stat : 20862 (march_hi) R 20857 20857 20607 0 -1 4202496 42598 0 0 0 59755 6 0 0 25 0 1 0 599533030 100675584 14100 33554432000 4194304 4331988 140736594304160 18446744073709551615 4228459 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/20862/statm: 24579 14100 216 34 0 23021 0
Current children cumulated CPU time (s) 4800.15
Current children cumulated vsize (KiB) 1685916

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 20857 and gives
#  childrusage.ru_utime.tv_sec=0
#  childrusage.ru_utime.tv_usec=2999
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=1999
# CPU time returned by wait4() is 0.004998
# while last known CPU time is 4800.15
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1201.71
CPU time (s): 4800.15
CPU user time (s): 4796.57
CPU system time (s): 3.58
CPU usage (%): 399.442
Max. virtual memory (cumulated for all children) (KiB): 4117184

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 0.002999
system time used= 0.001999
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 519
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 209
involuntary context switches= 6

runsolver used 1.93471 second user time and 4.6293 second system time

The end

Launcher Data 

Begin job on node118 at 2011-04-29 18:03:44
IDJOB=3291514
IDBENCH=84227
IDSOLVER=1612
FILE ID=node118/3291514-1304093023
RUNJOBID= node118-1304090617-20625
PBS_JOBID= 13170577
Free space on /tmp= 71604 MiB

SOLVER NAME= ppfolio par
BENCH NAME= SAT03/industrial/li/li-test4/li-test4-100.shuffled-as.sat03-370.cnf
COMMAND LINE= ppfolio BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3291514-1304093023/watcher-3291514-1304093023 -o /tmp/evaluation-result-3291514-1304093023/solver-3291514-1304093023 -C 4800 -W 1300 -M 15500  ppfolio HOME/instance-3291514-1304093023.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 5e5606ae7367b9272c60adc115594912
RANDOM SEED=478723925

node118.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.68
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.68
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5374.59
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      18138080 kB
Buffers:       1902260 kB
Cached:       12051132 kB
SwapCached:        600 kB
Active:        9767724 kB
Inactive:      4508644 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      18138080 kB
SwapTotal:    67111528 kB
SwapFree:     67110396 kB
Dirty:           13604 kB
Writeback:           0 kB
AnonPages:      321672 kB
Mapped:          16608 kB
Slab:           472692 kB
PageTables:       5468 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:  2312596 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71596 MiB
End job on node118 at 2011-04-29 18:23:47