Trace number 3190873

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ppfolio par? (TO) 4800.19 1201.71

General information on the benchmark

Namerandom/large/
unif-k3-r4.2-v20000-c84000-S493397373-056-UNKNOWN.cnf
MD5SUM53ba9a0375ae9f5fbde617f9de32bf14
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark35.2716
Satisfiable
(Un)Satisfiability was proved
Number of variables20000
Number of clauses84000
Sum of the clauses size252000
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 384000
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.00/0.00	This is pico-portfolio, a naive parallel portfolio
0.00/0.00	Copyright (C) 2011 Olivier Roussel
0.00/0.00	Using 4 processing units
0.00/0.00	This portfolio uses the following solvers:
0.00/0.00	[1] cryptominisat 2.7.1 (Mate Soos)
0.00/0.00	[2] clasp 1.3.6 (Martin Gebser, Benjamin Kaufmann, and Torsten Schaub)
0.00/0.00	[3] TNM 2009 (Wanxia Wei and Chu Min Li)
0.00/0.00	[4] lingeling 276-6264d55-100731 (Armin Biere)
0.00/0.00	[5] march_hi 2009 (Marijn Heule and Hans Van Maaren)
0.00/0.00	Starting cryptominisat 2.7.1 on core(s) 0
0.00/0.00	Starting clasp 1.3.6 on core(s) 4
0.00/0.00	Starting TNM 2009 on core(s) 6
0.00/0.00	Starting lingeling 276-6264d55-100731 on core(s) 2
0.00/0.00	Starting march_hi 2009 on core(s) 6

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3190873-1304040262/watcher-3190873-1304040262 -o /tmp/evaluation-result-3190873-1304040262/solver-3190873-1304040262 -C 4800 -W 1300 -M 15500 ppfolio HOME/instance-3190873-1304040262.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 10.23 9.95 9.47 8/182 11122
/proc/meminfo: memFree=13732704/32950928 swapFree=67111528/67111528
[pid=11122] ppid=11120 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/11122/stat : 11122 (ppfolio) R 11120 11122 10696 0 -1 4202496 521 0 0 0 0 0 0 0 25 0 1 0 515626030 18763776 389 33554432000 4194304 4287625 140733463884000 18446744073709551615 248089718287 0 0 4096 65536 0 0 0 17 4 0 0 0
/proc/11122/statm: 4581 389 341 23 0 91 0

[startup+0.431409 s]
/proc/loadavg: 10.23 9.95 9.47 8/182 11122
/proc/meminfo: memFree=13732704/32950928 swapFree=67111528/67111528
[pid=11122] ppid=11120 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/11122/stat : 11122 (ppfolio) S 11120 11122 10696 0 -1 4202496 521 0 0 0 0 0 0 0 23 0 1 0 515626030 18763776 389 33554432000 4194304 4287625 140733463884000 18446744073709551615 248089718287 0 0 4096 65536 3932055757922500607 0 0 17 4 0 0 0
/proc/11122/statm: 4581 389 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.500398 s]
/proc/loadavg: 10.23 9.95 9.47 8/182 11122
/proc/meminfo: memFree=13732704/32950928 swapFree=67111528/67111528
[pid=11122] ppid=11120 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/11122/stat : 11122 (ppfolio) S 11120 11122 10696 0 -1 4202496 521 0 0 0 0 0 0 0 23 0 1 0 515626030 18763776 389 33554432000 4194304 4287625 140733463884000 18446744073709551615 248089718287 0 0 4096 65536 3932055757922500607 0 0 17 4 0 0 0
/proc/11122/statm: 4581 389 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.600382 s]
/proc/loadavg: 10.23 9.95 9.47 8/182 11122
/proc/meminfo: memFree=13732704/32950928 swapFree=67111528/67111528
[pid=11122] ppid=11120 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/11122/stat : 11122 (ppfolio) S 11120 11122 10696 0 -1 4202496 521 0 0 0 0 0 0 0 23 0 1 0 515626030 18763776 389 33554432000 4194304 4287625 140733463884000 18446744073709551615 248089718287 0 0 4096 65536 3932055757922500607 0 0 17 4 0 0 0
/proc/11122/statm: 4581 389 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.700364 s]
/proc/loadavg: 10.23 9.95 9.47 8/182 11122
/proc/meminfo: memFree=13732704/32950928 swapFree=67111528/67111528
[pid=11122] ppid=11120 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/11122/stat : 11122 (ppfolio) S 11120 11122 10696 0 -1 4202496 521 0 0 0 0 0 0 0 23 0 1 0 515626030 18763776 389 33554432000 4194304 4287625 140733463884000 18446744073709551615 248089718287 0 0 4096 65536 3932055757922500607 0 0 17 4 0 0 0
/proc/11122/statm: 4581 389 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+1.50722 s]
/proc/loadavg: 10.23 9.95 9.47 11/188 11128
/proc/meminfo: memFree=13659868/32950928 swapFree=67111528/67111528
[pid=11122] ppid=11120 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/11122/stat : 11122 (ppfolio) S 11120 11122 10696 0 -1 4202496 521 0 0 0 0 0 0 0 18 0 1 0 515626030 18763776 389 33554432000 4194304 4287625 140733463884000 18446744073709551615 248089718287 0 0 4096 65536 3932055757922500607 0 0 17 4 0 0 0
/proc/11122/statm: 4581 389 341 23 0 91 0
[pid=11123] ppid=11122 vsize=42332 CPUtime=1.49 cores=0
/proc/11123/stat : 11123 (cryptominisat) R 11122 11122 10696 0 -1 4202496 7948 0 0 0 149 0 0 0 20 0 1 0 515626030 43347968 6311 33554432000 4194304 4587354 140733395550336 18446744073709551615 4248764 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/11123/statm: 10583 6311 326 96 0 7145 0
[pid=11124] ppid=11122 vsize=25768 CPUtime=1.49 cores=4
/proc/11124/stat : 11124 (clasp) R 11122 11122 10696 0 -1 4202496 4738 0 0 0 149 0 0 0 25 0 1 0 515626030 26386432 3773 33554432000 4194304 4785740 140736374708896 18446744073709551615 248089359045 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/11124/statm: 6442 3773 340 145 0 3487 0
[pid=11125] ppid=11122 vsize=459916 CPUtime=0.76 cores=6
/proc/11125/stat : 11125 (TNM) R 11122 11122 10696 0 -1 4202496 3486 0 0 0 76 0 0 0 25 0 1 0 515626031 470953984 3383 33554432000 134512640 134534564 4287408752 18446744073709551615 134528891 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/11125/statm: 114979 3383 95 6 0 113645 0
[pid=11126] ppid=11122 vsize=11424 CPUtime=1.49 cores=2
/proc/11126/stat : 11126 (lingeling) R 11122 11122 10696 0 -1 4202496 3518 0 0 0 149 0 0 0 20 0 1 0 515626031 11698176 1915 33554432000 4194304 4352236 140736091120320 18446744073709551615 4328026 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/11126/statm: 2856 1915 127 39 0 1936 0
[pid=11127] ppid=11122 vsize=21740 CPUtime=0.71 cores=6
/proc/11127/stat : 11127 (march_hi) R 11122 11122 10696 0 -1 4202496 19429 0 0 0 69 2 0 0 25 0 1 0 515626031 22261760 3178 33554432000 4194304 4331988 140735483309408 18446744073709551615 248089347188 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/11127/statm: 5435 3178 144 34 0 3877 0
Current children cumulated CPU time (s) 5.94
Current children cumulated vsize (KiB) 579504

[startup+3.10094 s]
/proc/loadavg: 10.23 9.95 9.47 11/188 11128
/proc/meminfo: memFree=13623908/32950928 swapFree=67111528/67111528
[pid=11122] ppid=11120 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/11122/stat : 11122 (ppfolio) S 11120 11122 10696 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 515626030 18763776 389 33554432000 4194304 4287625 140733463884000 18446744073709551615 248089718287 0 0 4096 65536 3932055757922500607 0 0 17 4 0 0 0
/proc/11122/statm: 4581 389 341 23 0 91 0
[pid=11123] ppid=11122 vsize=80016 CPUtime=3.09 cores=0
/proc/11123/stat : 11123 (cryptominisat) R 11122 11122 10696 0 -1 4202496 9101 0 0 0 308 1 0 0 25 0 1 0 515626030 81936384 7459 33554432000 4194304 4587354 140733395550336 18446744073709551615 4243943 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/11123/statm: 20004 7459 326 96 0 16566 0
[pid=11124] ppid=11122 vsize=34428 CPUtime=3.09 cores=4
/proc/11124/stat : 11124 (clasp) R 11122 11122 10696 0 -1 4202496 6882 0 0 0 308 1 0 0 25 0 1 0 515626030 35254272 5917 33554432000 4194304 4785740 140736374708896 18446744073709551615 4276894 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/11124/statm: 8607 5917 340 145 0 5652 0
[pid=11125] ppid=11122 vsize=459916 CPUtime=1.55 cores=6
/proc/11125/stat : 11125 (TNM) R 11122 11122 10696 0 -1 4202496 3486 0 0 0 155 0 0 0 25 0 1 0 515626031 470953984 3383 33554432000 134512640 134534564 4287408752 18446744073709551615 134528979 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/11125/statm: 114979 3383 95 6 0 113645 0
[pid=11126] ppid=11122 vsize=11424 CPUtime=3.09 cores=2
/proc/11126/stat : 11126 (lingeling) R 11122 11122 10696 0 -1 4202496 3518 0 0 0 309 0 0 0 25 0 1 0 515626031 11698176 1915 33554432000 4194304 4352236 140736091120320 18446744073709551615 4294498 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/11126/statm: 2856 1915 127 39 0 1936 0
[pid=11127] ppid=11122 vsize=46920 CPUtime=1.51 cores=6
/proc/11127/stat : 11127 (march_hi) R 11122 11122 10696 0 -1 4202496 28317 0 0 0 148 3 0 0 25 0 1 0 515626031 48046080 9303 33554432000 4194304 4331988 140735483309408 18446744073709551615 4263258 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/11127/statm: 11730 9303 173 34 0 10172 0
Current children cumulated CPU time (s) 12.33
Current children cumulated vsize (KiB) 651028

[startup+6.30037 s]
/proc/loadavg: 10.21 9.95 9.48 11/188 11128
/proc/meminfo: memFree=13587080/32950928 swapFree=67111528/67111528
[pid=11122] ppid=11120 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/11122/stat : 11122 (ppfolio) S 11120 11122 10696 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 515626030 18763776 389 33554432000 4194304 4287625 140733463884000 18446744073709551615 248089718287 0 0 4096 65536 3932055757922500607 0 0 17 4 0 0 0
/proc/11122/statm: 4581 389 341 23 0 91 0
[pid=11123] ppid=11122 vsize=80016 CPUtime=6.29 cores=0
/proc/11123/stat : 11123 (cryptominisat) R 11122 11122 10696 0 -1 4202496 11193 0 0 0 628 1 0 0 25 0 1 0 515626030 81936384 9540 33554432000 4194304 4587354 140733395550336 18446744073709551615 4238736 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/11123/statm: 20004 9540 326 96 0 16566 0
[pid=11124] ppid=11122 vsize=48252 CPUtime=6.28 cores=4
/proc/11124/stat : 11124 (clasp) R 11122 11122 10696 0 -1 4202496 10342 0 0 0 627 1 0 0 25 0 1 0 515626030 49410048 9377 33554432000 4194304 4785740 140736374708896 18446744073709551615 4553344 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/11124/statm: 12063 9377 340 145 0 9108 0
[pid=11125] ppid=11122 vsize=459916 CPUtime=3.13 cores=6
/proc/11125/stat : 11125 (TNM) R 11122 11122 10696 0 -1 4202496 3486 0 0 0 313 0 0 0 25 0 1 0 515626031 470953984 3383 33554432000 134512640 134534564 4287408752 18446744073709551615 134528877 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/11125/statm: 114979 3383 95 6 0 113645 0
[pid=11126] ppid=11122 vsize=14756 CPUtime=6.29 cores=2
/proc/11126/stat : 11126 (lingeling) R 11122 11122 10696 0 -1 4202496 4458 0 0 0 629 0 0 0 25 0 1 0 515626031 15110144 2855 33554432000 4194304 4352236 140736091120320 18446744073709551615 4328026 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/11126/statm: 3689 2855 127 39 0 2769 0
[pid=11127] ppid=11122 vsize=52124 CPUtime=3.12 cores=6
/proc/11127/stat : 11127 (march_hi) R 11122 11122 10696 0 -1 4202496 29350 0 0 0 308 4 0 0 25 0 1 0 515626031 53374976 10336 33554432000 4194304 4331988 140735483309408 18446744073709551615 4226434 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/11127/statm: 13031 10336 173 34 0 11473 0
Current children cumulated CPU time (s) 25.11
Current children cumulated vsize (KiB) 673388

[startup+12.7012 s]
/proc/loadavg: 10.19 9.95 9.48 11/188 11128
/proc/meminfo: memFree=13414256/32950928 swapFree=67111528/67111528
[pid=11122] ppid=11120 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/11122/stat : 11122 (ppfolio) S 11120 11122 10696 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 515626030 18763776 389 33554432000 4194304 4287625 140733463884000 18446744073709551615 248089718287 0 0 4096 65536 3932055757922500607 0 0 17 4 0 0 0
/proc/11122/statm: 4581 389 341 23 0 91 0
[pid=11123] ppid=11122 vsize=114340 CPUtime=12.69 cores=0
/proc/11123/stat : 11123 (cryptominisat) R 11122 11122 10696 0 -1 4202496 31890 0 0 0 1264 5 0 0 25 0 1 0 515626030 117084160 10261 33554432000 4194304 4587354 140733395550336 18446744073709551615 4238467 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/11123/statm: 28585 10261 337 96 0 25147 0
[pid=11124] ppid=11122 vsize=60200 CPUtime=12.69 cores=4
/proc/11124/stat : 11124 (clasp) R 11122 11122 10696 0 -1 4202496 13285 0 0 0 1266 3 0 0 25 0 1 0 515626030 61644800 12320 33554432000 4194304 4785740 140736374708896 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/11124/statm: 15050 12320 340 145 0 12095 0
[pid=11125] ppid=11122 vsize=459916 CPUtime=6.29 cores=6
/proc/11125/stat : 11125 (TNM) R 11122 11122 10696 0 -1 4202496 3486 0 0 0 629 0 0 0 25 0 1 0 515626031 470953984 3383 33554432000 134512640 134534564 4287408752 18446744073709551615 11385215 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/11125/statm: 114979 3383 95 6 0 113645 0
[pid=11126] ppid=11122 vsize=14756 CPUtime=12.69 cores=2
/proc/11126/stat : 11126 (lingeling) R 11122 11122 10696 0 -1 4202496 4458 0 0 0 1269 0 0 0 25 0 1 0 515626031 15110144 2855 33554432000 4194304 4352236 140736091120320 18446744073709551615 4327902 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/11126/statm: 3689 2855 127 39 0 2769 0
[pid=11127] ppid=11122 vsize=52784 CPUtime=6.32 cores=6
/proc/11127/stat : 11127 (march_hi) R 11122 11122 10696 0 -1 4202496 29530 0 0 0 628 4 0 0 25 0 1 0 515626031 54050816 10516 33554432000 4194304 4331988 140735483309408 18446744073709551615 4240763 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/11127/statm: 13196 10516 173 34 0 11638 0
Current children cumulated CPU time (s) 50.68
Current children cumulated vsize (KiB) 720320

[startup+25.501 s]
/proc/loadavg: 10.15 9.95 9.49 11/188 11129
/proc/meminfo: memFree=13373952/32950928 swapFree=67111528/67111528
[pid=11122] ppid=11120 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/11122/stat : 11122 (ppfolio) S 11120 11122 10696 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 515626030 18763776 389 33554432000 4194304 4287625 140733463884000 18446744073709551615 248089718287 0 0 4096 65536 3932055757922500607 0 0 17 4 0 0 0
/proc/11122/statm: 4581 389 341 23 0 91 0
[pid=11123] ppid=11122 vsize=59336 CPUtime=25.49 cores=0
/proc/11123/stat : 11123 (cryptominisat) R 11122 11122 10696 0 -1 4202496 50560 0 0 0 2538 11 0 0 25 0 1 0 515626030 60760064 9484 33554432000 4194304 4587354 140733395550336 18446744073709551615 4237679 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/11123/statm: 14834 9484 337 96 0 11396 0
[pid=11124] ppid=11122 vsize=73144 CPUtime=25.48 cores=4
/proc/11124/stat : 11124 (clasp) R 11122 11122 10696 0 -1 4202496 16515 0 0 0 2544 4 0 0 25 0 1 0 515626030 74899456 15550 33554432000 4194304 4785740 140736374708896 18446744073709551615 4562707 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/11124/statm: 18286 15550 340 145 0 15331 0
[pid=11125] ppid=11122 vsize=459916 CPUtime=12.69 cores=6
/proc/11125/stat : 11125 (TNM) R 11122 11122 10696 0 -1 4202496 3486 0 0 0 1269 0 0 0 25 0 1 0 515626031 470953984 3383 33554432000 134512640 134534564 4287408752 18446744073709551615 134528957 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/11125/statm: 114979 3383 95 6 0 113645 0
[pid=11126] ppid=11122 vsize=19424 CPUtime=25.48 cores=2
/proc/11126/stat : 11126 (lingeling) R 11122 11122 10696 0 -1 4202496 5625 0 0 0 2548 0 0 0 25 0 1 0 515626031 19890176 4022 33554432000 4194304 4352236 140736091120320 18446744073709551615 4328026 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/11126/statm: 4856 4022 127 39 0 3936 0
[pid=11127] ppid=11122 vsize=53576 CPUtime=12.65 cores=6
/proc/11127/stat : 11127 (march_hi) R 11122 11122 10696 0 -1 4202496 29735 0 0 0 1261 4 0 0 25 0 1 0 515626031 54861824 10721 33554432000 4194304 4331988 140735483309408 18446744073709551615 4211493 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/11127/statm: 13394 10721 173 34 0 11836 0
Current children cumulated CPU time (s) 101.79
Current children cumulated vsize (KiB) 683720

[startup+51.1005 s]
/proc/loadavg: 10.16 9.97 9.50 11/188 11129
/proc/meminfo: memFree=13530124/32950928 swapFree=67111528/67111528
[pid=11122] ppid=11120 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/11122/stat : 11122 (ppfolio) S 11120 11122 10696 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 515626030 18763776 389 33554432000 4194304 4287625 140733463884000 18446744073709551615 248089718287 0 0 4096 65536 3932055757922500607 0 0 17 4 0 0 0
/proc/11122/statm: 4581 389 341 23 0 91 0
[pid=11123] ppid=11122 vsize=890296 CPUtime=51.08 cores=0
/proc/11123/stat : 11123 (cryptominisat) R 11122 11122 10696 0 -1 4202496 97077 0 0 0 5088 20 0 0 25 0 1 0 515626030 911663104 12294 33554432000 4194304 4587354 140733395550336 18446744073709551615 4243984 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/11123/statm: 222574 12294 337 96 0 219136 0
[pid=11124] ppid=11122 vsize=93600 CPUtime=51.08 cores=4
/proc/11124/stat : 11124 (clasp) R 11122 11122 10696 0 -1 4202496 21513 0 0 0 5103 5 0 0 25 0 1 0 515626030 95846400 20548 33554432000 4194304 4785740 140736374708896 18446744073709551615 4553003 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/11124/statm: 23400 20548 340 145 0 20445 0
[pid=11125] ppid=11122 vsize=459916 CPUtime=25.39 cores=6
/proc/11125/stat : 11125 (TNM) R 11122 11122 10696 0 -1 4202496 3486 0 0 0 2539 0 0 0 25 0 1 0 515626031 470953984 3383 33554432000 134512640 134534564 4287408752 18446744073709551615 134528726 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/11125/statm: 114979 3383 95 6 0 113645 0
[pid=11126] ppid=11122 vsize=19424 CPUtime=51.08 cores=2

################
# More data... #
################

/proc/11127/statm: 14384 11708 173 34 0 12826 0
Current children cumulated CPU time (s) 3763.99
Current children cumulated vsize (KiB) 1750868

[startup+1002.3 s]
/proc/loadavg: 10.17 10.13 9.90 11/188 11188
/proc/meminfo: memFree=13190476/32950928 swapFree=67111528/67111528
[pid=11122] ppid=11120 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/11122/stat : 11122 (ppfolio) S 11120 11122 10696 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 515626030 18763776 389 33554432000 4194304 4287625 140733463884000 18446744073709551615 248089718287 0 0 4096 65536 3932055757922500607 0 0 17 4 0 0 0
/proc/11122/statm: 4581 389 341 23 0 91 0
[pid=11123] ppid=11122 vsize=919416 CPUtime=1002.31 cores=0
/proc/11123/stat : 11123 (cryptominisat) R 11122 11122 10696 0 -1 4202496 600605 0 0 0 100027 204 0 0 25 0 1 0 515626030 941481984 24202 33554432000 4194304 4587354 140733395550336 18446744073709551615 4238467 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/11123/statm: 229854 24202 337 96 0 226416 0
[pid=11124] ppid=11122 vsize=303644 CPUtime=1002.2 cores=4
/proc/11124/stat : 11124 (clasp) R 11122 11122 10696 0 -1 4202496 73229 0 0 0 100199 21 0 0 25 0 1 0 515626030 310931456 72264 33554432000 4194304 4785740 140736374708896 18446744073709551615 4287088 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/11124/statm: 75911 72264 340 145 0 72956 0
[pid=11125] ppid=11122 vsize=459916 CPUtime=498.41 cores=6
/proc/11125/stat : 11125 (TNM) R 11122 11122 10696 0 -1 4202496 3486 0 0 0 49841 0 0 0 25 0 1 0 515626031 470953984 3383 33554432000 134512640 134534564 4287408752 18446744073709551615 134525956 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/11125/statm: 114979 3383 95 6 0 113645 0
[pid=11126] ppid=11122 vsize=26180 CPUtime=1002.31 cores=2
/proc/11126/stat : 11126 (lingeling) R 11122 11122 10696 0 -1 4202496 9362 0 0 0 100230 1 0 0 25 0 1 0 515626031 26808320 5711 33554432000 4194304 4352236 140736091120320 18446744073709551615 4329086 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/11126/statm: 6545 5711 127 39 0 5625 0
[pid=11127] ppid=11122 vsize=57536 CPUtime=498.42 cores=6
/proc/11127/stat : 11127 (march_hi) R 11122 11122 10696 0 -1 4202496 30747 0 0 0 49837 5 0 0 25 0 1 0 515626031 58916864 11733 33554432000 4194304 4331988 140735483309408 18446744073709551615 4240214 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/11127/statm: 14384 11733 173 34 0 12826 0
Current children cumulated CPU time (s) 4003.65
Current children cumulated vsize (KiB) 1785016

[startup+1062.3 s]
/proc/loadavg: 10.17 10.13 9.91 11/188 11189
/proc/meminfo: memFree=13168856/32950928 swapFree=67111528/67111528
[pid=11122] ppid=11120 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/11122/stat : 11122 (ppfolio) S 11120 11122 10696 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 515626030 18763776 389 33554432000 4194304 4287625 140733463884000 18446744073709551615 248089718287 0 0 4096 65536 3932055757922500607 0 0 17 4 0 0 0
/proc/11122/statm: 4581 389 341 23 0 91 0
[pid=11123] ppid=11122 vsize=923064 CPUtime=1062.31 cores=0
/proc/11123/stat : 11123 (cryptominisat) R 11122 11122 10696 0 -1 4202496 624791 0 0 0 106016 215 0 0 25 0 1 0 515626030 945217536 21115 33554432000 4194304 4587354 140733395550336 18446744073709551615 4238149 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/11123/statm: 230766 21115 337 96 0 227328 0
[pid=11124] ppid=11122 vsize=303892 CPUtime=1062.2 cores=4
/proc/11124/stat : 11124 (clasp) R 11122 11122 10696 0 -1 4202496 73314 0 0 0 106199 21 0 0 25 0 1 0 515626030 311185408 72349 33554432000 4194304 4785740 140736374708896 18446744073709551615 4553003 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/11124/statm: 75973 72349 340 145 0 73018 0
[pid=11125] ppid=11122 vsize=459916 CPUtime=528.27 cores=6
/proc/11125/stat : 11125 (TNM) R 11122 11122 10696 0 -1 4202496 3486 0 0 0 52827 0 0 0 25 0 1 0 515626031 470953984 3383 33554432000 134512640 134534564 4287408752 18446744073709551615 134528979 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/11125/statm: 114979 3383 95 6 0 113645 0
[pid=11126] ppid=11122 vsize=26180 CPUtime=1062.31 cores=2
/proc/11126/stat : 11126 (lingeling) R 11122 11122 10696 0 -1 4202496 9874 0 0 0 106230 1 0 0 25 0 1 0 515626031 26808320 5711 33554432000 4194304 4352236 140736091120320 18446744073709551615 4276778 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/11126/statm: 6545 5711 127 39 0 5625 0
[pid=11127] ppid=11122 vsize=57668 CPUtime=528.23 cores=6
/proc/11127/stat : 11127 (march_hi) R 11122 11122 10696 0 -1 4202496 30764 0 0 0 52818 5 0 0 25 0 1 0 515626031 59052032 11750 33554432000 4194304 4331988 140735483309408 18446744073709551615 4211509 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/11127/statm: 14417 11750 173 34 0 12859 0
Current children cumulated CPU time (s) 4243.32
Current children cumulated vsize (KiB) 1789044

[startup+1122.3 s]
/proc/loadavg: 10.23 10.15 9.93 11/188 11191
/proc/meminfo: memFree=13140316/32950928 swapFree=67111528/67111528
[pid=11122] ppid=11120 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/11122/stat : 11122 (ppfolio) S 11120 11122 10696 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 515626030 18763776 389 33554432000 4194304 4287625 140733463884000 18446744073709551615 248089718287 0 0 4096 65536 3932055757922500607 0 0 17 4 0 0 0
/proc/11122/statm: 4581 389 341 23 0 91 0
[pid=11123] ppid=11122 vsize=137156 CPUtime=1122.32 cores=0
/proc/11123/stat : 11123 (cryptominisat) R 11122 11122 10696 0 -1 4202496 646431 0 0 0 112006 226 0 0 25 0 1 0 515626030 140447744 26794 33554432000 4194304 4587354 140733395550336 18446744073709551615 4228756 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/11123/statm: 34289 26794 337 96 0 30851 0
[pid=11124] ppid=11122 vsize=318484 CPUtime=1122.21 cores=4
/proc/11124/stat : 11124 (clasp) R 11122 11122 10696 0 -1 4202496 76873 0 0 0 112198 23 0 0 25 0 1 0 515626030 326127616 75908 33554432000 4194304 4785740 140736374708896 18446744073709551615 4278587 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/11124/statm: 79621 75908 340 145 0 76666 0
[pid=11125] ppid=11122 vsize=459916 CPUtime=558.1 cores=6
/proc/11125/stat : 11125 (TNM) R 11122 11122 10696 0 -1 4202496 3486 0 0 0 55810 0 0 0 25 0 1 0 515626031 470953984 3383 33554432000 134512640 134534564 4287408752 18446744073709551615 134530551 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/11125/statm: 114979 3383 95 6 0 113645 0
[pid=11126] ppid=11122 vsize=26180 CPUtime=1122.32 cores=2
/proc/11126/stat : 11126 (lingeling) R 11122 11122 10696 0 -1 4202496 10386 0 0 0 112231 1 0 0 25 0 1 0 515626031 26808320 5711 33554432000 4194304 4352236 140736091120320 18446744073709551615 4331578 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/11126/statm: 6545 5711 127 39 0 5625 0
[pid=11127] ppid=11122 vsize=57668 CPUtime=558.08 cores=6
/proc/11127/stat : 11127 (march_hi) R 11122 11122 10696 0 -1 4202496 30781 0 0 0 55803 5 0 0 25 0 1 0 515626031 59052032 11767 33554432000 4194304 4331988 140735483309408 18446744073709551615 4228970 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/11127/statm: 14417 11767 173 34 0 12859 0
Current children cumulated CPU time (s) 4483.03
Current children cumulated vsize (KiB) 1017728

[startup+1182.3 s]
/proc/loadavg: 10.25 10.17 9.95 11/188 11193
/proc/meminfo: memFree=13102972/32950928 swapFree=67111528/67111528
[pid=11122] ppid=11120 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/11122/stat : 11122 (ppfolio) S 11120 11122 10696 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 515626030 18763776 389 33554432000 4194304 4287625 140733463884000 18446744073709551615 248089718287 0 0 4096 65536 3932055757922500607 0 0 17 4 0 0 0
/proc/11122/statm: 4581 389 341 23 0 91 0
[pid=11123] ppid=11122 vsize=921120 CPUtime=1182.31 cores=0
/proc/11123/stat : 11123 (cryptominisat) R 11122 11122 10696 0 -1 4202496 670533 0 0 0 117992 239 0 0 25 0 1 0 515626030 943226880 22296 33554432000 4194304 4587354 140733395550336 18446744073709551615 4238522 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/11123/statm: 230280 22296 337 96 0 226842 0
[pid=11124] ppid=11122 vsize=343624 CPUtime=1182.19 cores=4
/proc/11124/stat : 11124 (clasp) R 11122 11122 10696 0 -1 4202496 83156 0 0 0 118195 24 0 0 25 0 1 0 515626030 351870976 82191 33554432000 4194304 4785740 140736374708896 18446744073709551615 4309555 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/11124/statm: 85906 82191 340 145 0 82951 0
[pid=11125] ppid=11122 vsize=459916 CPUtime=587.94 cores=6
/proc/11125/stat : 11125 (TNM) R 11122 11122 10696 0 -1 4202496 3486 0 0 0 58793 1 0 0 25 0 1 0 515626031 470953984 3383 33554432000 134512640 134534564 4287408752 18446744073709551615 134529031 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/11125/statm: 114979 3383 95 6 0 113645 0
[pid=11126] ppid=11122 vsize=26180 CPUtime=1182.31 cores=2
/proc/11126/stat : 11126 (lingeling) R 11122 11122 10696 0 -1 4202496 10898 0 0 0 118230 1 0 0 25 0 1 0 515626031 26808320 5711 33554432000 4194304 4352236 140736091120320 18446744073709551615 4328423 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/11126/statm: 6545 5711 127 39 0 5625 0
[pid=11127] ppid=11122 vsize=57800 CPUtime=587.92 cores=6
/proc/11127/stat : 11127 (march_hi) R 11122 11122 10696 0 -1 4202496 30796 0 0 0 58787 5 0 0 25 0 1 0 515626031 59187200 11782 33554432000 4194304 4331988 140735483309408 18446744073709551615 4226440 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/11127/statm: 14450 11782 173 34 0 12892 0
Current children cumulated CPU time (s) 4722.67
Current children cumulated vsize (KiB) 1826964



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1201.7 s]
/proc/loadavg: 10.18 10.15 9.95 11/188 11193
/proc/meminfo: memFree=13125412/32950928 swapFree=67111528/67111528
[pid=11122] ppid=11120 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/11122/stat : 11122 (ppfolio) S 11120 11122 10696 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 515626030 18763776 389 33554432000 4194304 4287625 140733463884000 18446744073709551615 248089718287 0 0 4096 65536 3932055757922500607 0 0 17 4 0 0 0
/proc/11122/statm: 4581 389 341 23 0 91 0
[pid=11123] ppid=11122 vsize=913748 CPUtime=1201.72 cores=0
/proc/11123/stat : 11123 (cryptominisat) R 11122 11122 10696 0 -1 4202496 679893 0 0 0 119929 243 0 0 25 0 1 0 515626030 935677952 19005 33554432000 4194304 4587354 140733395550336 18446744073709551615 4230359 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/11123/statm: 228437 19005 337 96 0 224999 0
[pid=11124] ppid=11122 vsize=343624 CPUtime=1201.6 cores=4
/proc/11124/stat : 11124 (clasp) R 11122 11122 10696 0 -1 4202496 83161 0 0 0 120136 24 0 0 25 0 1 0 515626030 351870976 82196 33554432000 4194304 4785740 140736374708896 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/11124/statm: 85906 82196 340 145 0 82951 0
[pid=11125] ppid=11122 vsize=459916 CPUtime=597.6 cores=6
/proc/11125/stat : 11125 (TNM) R 11122 11122 10696 0 -1 4202496 3486 0 0 0 59759 1 0 0 25 0 1 0 515626031 470953984 3383 33554432000 134512640 134534564 4287408752 18446744073709551615 134528511 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/11125/statm: 114979 3383 95 6 0 113645 0
[pid=11126] ppid=11122 vsize=26180 CPUtime=1201.72 cores=2
/proc/11126/stat : 11126 (lingeling) R 11122 11122 10696 0 -1 4202496 10898 0 0 0 120171 1 0 0 25 0 1 0 515626031 26808320 5711 33554432000 4194304 4352236 140736091120320 18446744073709551615 4328039 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/11126/statm: 6545 5711 127 39 0 5625 0
[pid=11127] ppid=11122 vsize=57800 CPUtime=597.55 cores=6
/proc/11127/stat : 11127 (march_hi) R 11122 11122 10696 0 -1 4202496 30802 0 0 0 59750 5 0 0 25 0 1 0 515626031 59187200 11788 33554432000 4194304 4331988 140735483309408 18446744073709551615 4265367 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/11127/statm: 14450 11788 173 34 0 12892 0
Current children cumulated CPU time (s) 4800.19
Current children cumulated vsize (KiB) 1819592

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1201.7 s]
/proc/loadavg: 10.18 10.15 9.95 11/188 11193
/proc/meminfo: memFree=13125412/32950928 swapFree=67111528/67111528
[pid=11122] ppid=11120 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/11122/stat : 11122 (ppfolio) S 11120 11122 10696 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 515626030 18763776 389 33554432000 4194304 4287625 140733463884000 18446744073709551615 248089718287 0 0 4096 65536 3932055757922500607 0 0 17 4 0 0 0
/proc/11122/statm: 4581 389 341 23 0 91 0
[pid=11123] ppid=11122 vsize=913748 CPUtime=1201.72 cores=0
/proc/11123/stat : 11123 (cryptominisat) R 11122 11122 10696 0 -1 4202496 679893 0 0 0 119929 243 0 0 25 0 1 0 515626030 935677952 19005 33554432000 4194304 4587354 140733395550336 18446744073709551615 4230359 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/11123/statm: 228437 19005 337 96 0 224999 0
[pid=11124] ppid=11122 vsize=343624 CPUtime=1201.6 cores=4
/proc/11124/stat : 11124 (clasp) R 11122 11122 10696 0 -1 4202496 83161 0 0 0 120136 24 0 0 25 0 1 0 515626030 351870976 82196 33554432000 4194304 4785740 140736374708896 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/11124/statm: 85906 82196 340 145 0 82951 0
[pid=11125] ppid=11122 vsize=459916 CPUtime=597.6 cores=6
/proc/11125/stat : 11125 (TNM) R 11122 11122 10696 0 -1 4202496 3486 0 0 0 59759 1 0 0 25 0 1 0 515626031 470953984 3383 33554432000 134512640 134534564 4287408752 18446744073709551615 134528511 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/11125/statm: 114979 3383 95 6 0 113645 0
[pid=11126] ppid=11122 vsize=26180 CPUtime=1201.72 cores=2
/proc/11126/stat : 11126 (lingeling) R 11122 11122 10696 0 -1 4202496 10898 0 0 0 120171 1 0 0 25 0 1 0 515626031 26808320 5711 33554432000 4194304 4352236 140736091120320 18446744073709551615 4328039 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/11126/statm: 6545 5711 127 39 0 5625 0
[pid=11127] ppid=11122 vsize=57800 CPUtime=597.55 cores=6
/proc/11127/stat : 11127 (march_hi) R 11122 11122 10696 0 -1 4202496 30802 0 0 0 59750 5 0 0 25 0 1 0 515626031 59187200 11788 33554432000 4194304 4331988 140735483309408 18446744073709551615 4265367 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/11127/statm: 14450 11788 173 34 0 12892 0
Current children cumulated CPU time (s) 4800.19
Current children cumulated vsize (KiB) 1819592

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 11122 and gives
#  childrusage.ru_utime.tv_sec=3596
#  childrusage.ru_utime.tv_usec=123306
#  childrusage.ru_stime.tv_sec=2
#  childrusage.ru_stime.tv_usec=532614
# CPU time returned by wait4() is 3598.66
# while last known CPU time is 4800.19
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1201.71
CPU time (s): 4800.19
CPU user time (s): 4797.45
CPU system time (s): 2.74
CPU usage (%): 399.445
Max. virtual memory (cumulated for all children) (KiB): 2701372

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 3596.12
system time used= 2.53261
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 725603
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 288
involuntary context switches= 30860

runsolver used 1.77773 second user time and 4.78527 second system time

The end

Launcher Data 

Begin job on node123 at 2011-04-29 03:24:22
IDJOB=3190873
IDBENCH=82604
IDSOLVER=1612
FILE ID=node123/3190873-1304040262
RUNJOBID= node123-1304037296-10714
PBS_JOBID= 13170370
Free space on /tmp= 72804 MiB

SOLVER NAME= ppfolio par
BENCH NAME= SAT11/random/large/unif-k3-r4.2-v20000-c84000-S493397373-056-UNKNOWN.cnf
COMMAND LINE= ppfolio BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3190873-1304040262/watcher-3190873-1304040262 -o /tmp/evaluation-result-3190873-1304040262/solver-3190873-1304040262 -C 4800 -W 1300 -M 15500  ppfolio HOME/instance-3190873-1304040262.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 53ba9a0375ae9f5fbde617f9de32bf14
RANDOM SEED=1241021494

node123.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      13733108 kB
Buffers:       2191548 kB
Cached:       16024300 kB
SwapCached:          0 kB
Active:        9489908 kB
Inactive:      9178260 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      13733108 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:           11700 kB
Writeback:           0 kB
AnonPages:      451184 kB
Mapped:          16944 kB
Slab:           486204 kB
PageTables:       5516 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:  2794504 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72792 MiB
End job on node123 at 2011-04-29 03:44:26