Trace number 3210141

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
SAT09 reference solver: march_hi hi? (TO) 5000.05 4999.93

General information on the benchmark

Namerandom/large/
unif-k7-r85-v150-c12750-S1290198737-013-UNKNOWN.cnf
MD5SUM707c4bcddcda116edc4a57a77dc3ba68
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark64.6052
Satisfiable
(Un)Satisfiability was proved
Number of variables150
Number of clauses12750
Sum of the clauses size89250
Maximum clause length7
Minimum clause length7
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 512750

Solver Data 

0.79/0.87	c main():: ***                                   [ march satisfiability solver ]                                   ***
0.79/0.87	c main()::  **                Copyright (C) 2001-2009 M.J.H. Heule, J.E. van Zwieten, and M. Dufour                 **
0.79/0.87	c main()::   *  This program may be redistributed and/or modified under the terms of the GNU Gereral Public License  *
0.79/0.87	c main()::
0.79/0.87	c initFormula():: searching for DIMACS p-line....
0.79/0.87	c full lookahead due to high density!
0.79/0.87	c initFormula():: the DIMACS p-line indicates a CNF of 150 variables and 12750 clauses.
0.79/0.87	c parseCNF():: parsing....
0.79/0.87	c parseCNF():: the CNF contains 0 unary clauses.
0.79/0.87	c runParser():: parsing was successful, warming up engines...
0.79/0.87	c simplify_formula():: removed 0 tautological, 0 satisfied and 0 duplicate clauses
0.79/0.87	c preprocessing fase I completed:: there are now 150 free variables and 12750 clauses.
0.79/0.87	c stat :: sign balance is biased by 0.007
0.79/0.87	c using k-SAT heuristics (size based diff)
0.79/0.87	c simplify_formula():: removed 0 tautological, 0 satisfied and 0 duplicate clauses
0.79/0.87	c simplify_formula():: removed 0 tautological, 0 satisfied and 0 duplicate clauses
0.79/0.87	c main():: clause / variable ratio: ( 12750 / 150 ) = 85.00
0.79/0.87	c longest clause has size 7
0.79/0.87	c simplify_formula():: removed 0 tautological, 0 satisfied and 0 duplicate clauses
0.79/0.87	c number of free variables = 150
0.79/0.87	c dynamic_preselect_setsize :: off
0.79/0.87	c main():: all systems go!
0.79/0.87	c |----------------------------------------------------------------|
c |*
c |**
c |****
c |********
c |****************
c |********************************
c |****************************************************************| ( 0/999) NodeCount: 175
0.89/0.94	c |----------------------------------------------------------------|
c |*
c |**
c |****
c |********
c |****************
c |********************************
c |*********************************
c |**********************************
c |************************************
c |****************************************
c |************************************************
c |*************************************************
c |**************************************************
c |****************************************************
c |********************************************************
c |*********************************************************
c |**********************************************************
c |************************************************************
c |*************************************************************
c |**************************************************************
c |***************************************************************
c |****************************************************************| ( 1/999) NodeCount: 2287
1.39/1.41	c |----------------------------------------------------------------|
c |*
c |**
c |****
c |********
c |****************
c |*****************
c |******************
c |********************
c |************************
c |*************************
c |**************************
c |****************************
c |*****************************
c |******************************
c |*******************************
c |********************************
c |*********************************
c |**********************************
c |************************************
c |****************************************
c |*****************************************
c |******************************************
c |********************************************
c |*********************************************
c |**********************************************
c |***********************************************
c |************************************************
c |*************************************************
c |**************************************************
c |****************************************************
c |*****************************************************
c |******************************************************
c |*******************************************************
c |********************************************************
c |*********************************************************
c |**********************************************************
c |***********************************************************
c |************************************************************
c |*************************************************************
c |**************************************************************
c |***************************************************************
c |****************************************************************| ( 2/999) NodeCount: 45321
9.69/9.78	c |----------------------------------------------------------------|
c |*
c |**
c |****
c |********
c |*********
c |**********
c |************
c |*************
c |**************
c |***************
c |****************
c |*****************
c |******************
c |********************
c |*********************
c |**********************
c |***********************
c |************************
c |*************************
c |**************************
c |***************************
c |****************************
c |*****************************
c |******************************
c |*******************************
c |********************************
c |*********************************
c |**********************************
c |************************************
c |*************************************
c |**************************************
c |***************************************
c |****************************************
c |*****************************************
c |******************************************
c |*******************************************
c |********************************************
c |*********************************************
c |**********************************************
c |***********************************************
c |************************************************
c |*************************************************
c |**************************************************
c |***************************************************
c |****************************************************
c |*****************************************************
c |******************************************************
c |*******************************************************
c |********************************************************
c |*********************************************************
c |**********************************************************
c |***********************************************************
c |************************************************************
c |*************************************************************
c |**************************************************************
c |***************************************************************
c |****************************************************************| ( 3/999) NodeCount: 684126
130.99/131.03	c |----------------------------------------------------------------|
c |*
c |**
c |****
c |*****
c |******
c |*******
c |********
c |*********
c |**********
c |***********
c |************
c |*************
c |**************
c |***************
c |****************
c |*****************
c |******************
c |*******************
c |********************
c |*********************
c |**********************
c |***********************
c |************************
c |*************************
c |**************************
c |***************************
c |****************************
c |*****************************
c |******************************
c |*******************************
c |********************************
c |*********************************
c |**********************************
c |***********************************
c |************************************
c |*************************************
c |**************************************
c |***************************************
c |****************************************
c |*****************************************
c |******************************************
c |*******************************************
c |********************************************
c |*********************************************
c |**********************************************
c |***********************************************
c |************************************************
c |*************************************************
c |**************************************************
c |***************************************************
c |****************************************************
c |*****************************************************
c |******************************************************
c |*******************************************************
c |********************************************************
c |*********************************************************
c |**********************************************************
c |***********************************************************
c |************************************************************
c |*************************************************************
c |**************************************************************
c |***************************************************************
c |****************************************************************| ( 4/999) NodeCount: 8000284
1540.23/1540.20	c |----------------------------------------------------------------|
c |*
c |**
c |***
c |****
c |*****
c |******
c |*******
c |********
c |*********
c |**********
c |***********
c |************
c |*************
c |**************
c |***************
c |****************
c |*****************
c |******************
c |*******************
c |********************
c |*********************
c |**********************
c |***********************
c |************************
c |*************************
c |**************************
c |***************************
c |****************************
c |*****************************
c |******************************
c |*******************************
c |********************************
c |*********************************
c |**********************************
c |***********************************
c |************************************
c |*************************************
c |**************************************
c |***************************************
c |****************************************
c |*****************************************
c |******************************************
c |*******************************************
c |********************************************
c |*********************************************
c |**********************************************
c |***********************************************
c |************************************************
c |*************************************************
c |**************************************************
c |***************************************************
c |****************************************************
c |*****************************************************
c |******************************************************
c |*******************************************************

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3210141-1303152052/watcher-3210141-1303152052 -o /tmp/evaluation-result-3210141-1303152052/solver-3210141-1303152052 -C 5000 -W 5000 -M 15500 HOME/march_hi HOME/instance-3210141-1303152052.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 5000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 5030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5000 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 1.70 1.94 1.98 2/177 21206
/proc/meminfo: memFree=17294004/32951124 swapFree=67102128/67111528
[pid=21206] ppid=21204 vsize=1048 CPUtime=0 cores=1,3,5,7
/proc/21206/stat : 21206 (march_hi) R 21204 21206 21168 0 -1 4202496 165 0 0 0 0 0 0 0 25 0 1 0 505435882 1073152 121 33554432000 134512640 135197378 4294522576 18446744073709551615 134697414 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21206/statm: 262 121 58 168 0 93 0

[startup+0.0783411 s]
/proc/loadavg: 1.70 1.94 1.98 2/177 21206
/proc/meminfo: memFree=17294004/32951124 swapFree=67102128/67111528
[pid=21206] ppid=21204 vsize=2936 CPUtime=0.07 cores=1,3,5,7
/proc/21206/stat : 21206 (march_hi) R 21204 21206 21168 0 -1 4202496 2131 0 0 0 7 0 0 0 25 0 1 0 505435882 3006464 535 33554432000 134512640 135197378 4294522576 18446744073709551615 134578126 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21206/statm: 734 535 82 168 0 565 0
Current children cumulated CPU time (s) 0.07
Current children cumulated vsize (KiB) 2936

[startup+0.100333 s]
/proc/loadavg: 1.70 1.94 1.98 2/177 21206
/proc/meminfo: memFree=17294004/32951124 swapFree=67102128/67111528
[pid=21206] ppid=21204 vsize=2936 CPUtime=0.09 cores=1,3,5,7
/proc/21206/stat : 21206 (march_hi) R 21204 21206 21168 0 -1 4202496 2131 0 0 0 9 0 0 0 25 0 1 0 505435882 3006464 535 33554432000 134512640 135197378 4294522576 18446744073709551615 134578096 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21206/statm: 734 535 82 168 0 565 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 2936

[startup+0.300308 s]
/proc/loadavg: 1.70 1.94 1.98 2/177 21206
/proc/meminfo: memFree=17294004/32951124 swapFree=67102128/67111528
[pid=21206] ppid=21204 vsize=2936 CPUtime=0.29 cores=1,3,5,7
/proc/21206/stat : 21206 (march_hi) R 21204 21206 21168 0 -1 4202496 2131 0 0 0 29 0 0 0 25 0 1 0 505435882 3006464 535 33554432000 134512640 135197378 4294522576 18446744073709551615 134578135 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21206/statm: 734 535 82 168 0 565 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 2936

[startup+0.700238 s]
/proc/loadavg: 1.70 1.94 1.98 2/177 21206
/proc/meminfo: memFree=17294004/32951124 swapFree=67102128/67111528
[pid=21206] ppid=21204 vsize=2936 CPUtime=0.69 cores=1,3,5,7
/proc/21206/stat : 21206 (march_hi) R 21204 21206 21168 0 -1 4202496 2131 0 0 0 69 0 0 0 25 0 1 0 505435882 3006464 535 33554432000 134512640 135197378 4294522576 18446744073709551615 134578117 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21206/statm: 734 535 82 168 0 565 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 2936

[startup+1.5011 s]
/proc/loadavg: 1.70 1.94 1.98 3/178 21207
/proc/meminfo: memFree=17289104/32951124 swapFree=67102128/67111528
[pid=21206] ppid=21204 vsize=4716 CPUtime=1.49 cores=1,3,5,7
/proc/21206/stat : 21206 (march_hi) R 21204 21206 21168 0 -1 4202496 2650 0 0 0 149 0 0 0 25 0 1 0 505435882 4829184 866 33554432000 134512640 135197378 4294522576 18446744073709551615 134628249 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21206/statm: 1179 866 93 168 0 1010 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 4716

[startup+3.10082 s]
/proc/loadavg: 1.70 1.94 1.98 3/178 21207
/proc/meminfo: memFree=17288484/32951124 swapFree=67102128/67111528
[pid=21206] ppid=21204 vsize=4716 CPUtime=3.09 cores=1,3,5,7
/proc/21206/stat : 21206 (march_hi) R 21204 21206 21168 0 -1 4202496 2650 0 0 0 309 0 0 0 25 0 1 0 505435882 4829184 866 33554432000 134512640 135197378 4294522576 18446744073709551615 134524890 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21206/statm: 1179 866 93 168 0 1010 0
Current children cumulated CPU time (s) 3.09
Current children cumulated vsize (KiB) 4716

[startup+6.30026 s]
/proc/loadavg: 1.72 1.94 1.98 3/178 21207
/proc/meminfo: memFree=17288068/32951124 swapFree=67102128/67111528
[pid=21206] ppid=21204 vsize=5400 CPUtime=6.29 cores=1,3,5,7
/proc/21206/stat : 21206 (march_hi) R 21204 21206 21168 0 -1 4202496 2804 0 0 0 629 0 0 0 25 0 1 0 505435882 5529600 1020 33554432000 134512640 135197378 4294522576 18446744073709551615 134548733 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21206/statm: 1350 1020 93 168 0 1181 0
Current children cumulated CPU time (s) 6.29
Current children cumulated vsize (KiB) 5400

[startup+12.7001 s]
/proc/loadavg: 1.75 1.94 1.98 3/178 21207
/proc/meminfo: memFree=17286612/32951124 swapFree=67102128/67111528
[pid=21206] ppid=21204 vsize=6252 CPUtime=12.69 cores=1,3,5,7
/proc/21206/stat : 21206 (march_hi) R 21204 21206 21168 0 -1 4202496 2998 0 0 0 1269 0 0 0 25 0 1 0 505435882 6402048 1214 33554432000 134512640 135197378 4294522576 18446744073709551615 134550648 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21206/statm: 1563 1214 93 168 0 1394 0
Current children cumulated CPU time (s) 12.69
Current children cumulated vsize (KiB) 6252

[startup+25.5009 s]
/proc/loadavg: 1.80 1.94 1.98 3/178 21207
/proc/meminfo: memFree=17284184/32951124 swapFree=67102128/67111528
[pid=21206] ppid=21204 vsize=8168 CPUtime=25.49 cores=1,3,5,7
/proc/21206/stat : 21206 (march_hi) R 21204 21206 21168 0 -1 4202496 3386 0 0 0 2549 0 0 0 25 0 1 0 505435882 8364032 1602 33554432000 134512640 135197378 4294522576 18446744073709551615 134603175 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21206/statm: 2042 1602 93 168 0 1873 0
Current children cumulated CPU time (s) 25.49
Current children cumulated vsize (KiB) 8168

[startup+51.1004 s]
/proc/loadavg: 1.87 1.95 1.98 3/178 21208
/proc/meminfo: memFree=17278276/32951124 swapFree=67102128/67111528
[pid=21206] ppid=21204 vsize=12480 CPUtime=51.09 cores=1,3,5,7
/proc/21206/stat : 21206 (march_hi) R 21204 21206 21168 0 -1 4202496 4175 0 0 0 5109 0 0 0 25 0 1 0 505435882 12779520 2391 33554432000 134512640 135197378 4294522576 18446744073709551615 134550665 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21206/statm: 3120 2391 93 168 0 2951 0
Current children cumulated CPU time (s) 51.09
Current children cumulated vsize (KiB) 12480

[startup+102.306 s]
/proc/loadavg: 1.94 1.95 1.98 3/178 21209
/proc/meminfo: memFree=17265876/32951124 swapFree=67102128/67111528
[pid=21206] ppid=21204 vsize=22180 CPUtime=102.3 cores=1,3,5,7
/proc/21206/stat : 21206 (march_hi) R 21204 21206 21168 0 -1 4202496 5785 0 0 0 10229 1 0 0 25 0 1 0 505435882 22712320 4001 33554432000 134512640 135197378 4294522576 18446744073709551615 134550709 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21206/statm: 5545 4001 93 168 0 5376 0
Current children cumulated CPU time (s) 102.3
Current children cumulated vsize (KiB) 22180

[startup+162.301 s]
/proc/loadavg: 1.98 1.96 1.98 3/178 21211
/proc/meminfo: memFree=17252336/32951124 swapFree=67102128/67111528
[pid=21206] ppid=21204 vsize=30912 CPUtime=162.29 cores=1,3,5,7
/proc/21206/stat : 21206 (march_hi) R 21204 21206 21168 0 -1 4202496 7610 0 0 0 16228 1 0 0 25 0 1 0 505435882 31653888 5826 33554432000 134512640 135197378 4294522576 18446744073709551615 134550677 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21206/statm: 7728 5826 93 168 0 7559 0
Current children cumulated CPU time (s) 162.29
Current children cumulated vsize (KiB) 30912

[startup+222.3 s]
/proc/loadavg: 1.99 1.96 1.98 3/178 21212
/proc/meminfo: memFree=17238296/32951124 swapFree=67102128/67111528
[pid=21206] ppid=21204 vsize=44008 CPUtime=222.29 cores=1,3,5,7
/proc/21206/stat : 21206 (march_hi) R 21204 21206 21168 0 -1 4202496 9485 0 0 0 22228 1 0 0 25 0 1 0 505435882 45064192 7701 33554432000 134512640 135197378 4294522576 18446744073709551615 134550709 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21206/statm: 11002 7701 93 168 0 10833 0
Current children cumulated CPU time (s) 222.29
Current children cumulated vsize (KiB) 44008

[startup+282.301 s]
/proc/loadavg: 1.99 1.97 1.98 3/178 21214
/proc/meminfo: memFree=17224488/32951124 swapFree=67102128/67111528
[pid=21206] ppid=21204 vsize=44816 CPUtime=282.3 cores=1,3,5,7
/proc/21206/stat : 21206 (march_hi) R 21204 21206 21168 0 -1 4202496 11406 0 0 0 28228 2 0 0 25 0 1 0 505435882 45891584 9622 33554432000 134512640 135197378 4294522576 18446744073709551615 134539117 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21206/statm: 11204 9622 93 168 0 11035 0
Current children cumulated CPU time (s) 282.3
Current children cumulated vsize (KiB) 44816

[startup+342.3 s]
/proc/loadavg: 1.99 1.97 1.98 3/178 21216
/proc/meminfo: memFree=17210352/32951124 swapFree=67102128/67111528
[pid=21206] ppid=21204 vsize=64460 CPUtime=342.3 cores=1,3,5,7
/proc/21206/stat : 21206 (march_hi) R 21204 21206 21168 0 -1 4202496 13284 0 0 0 34228 2 0 0 25 0 1 0 505435882 66007040 11500 33554432000 134512640 135197378 4294522576 18446744073709551615 134603270 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21206/statm: 16115 11500 93 168 0 15946 0
Current children cumulated CPU time (s) 342.3
Current children cumulated vsize (KiB) 64460

[startup+402.301 s]
/proc/loadavg: 1.99 1.97 1.98 3/178 21217
/proc/meminfo: memFree=17196684/32951124 swapFree=67102128/67111528
[pid=21206] ppid=21204 vsize=64460 CPUtime=402.3 cores=1,3,5,7
/proc/21206/stat : 21206 (march_hi) R 21204 21206 21168 0 -1 4202496 15146 0 0 0 40228 2 0 0 25 0 1 0 505435882 66007040 13362 33554432000 134512640 135197378 4294522576 18446744073709551615 134526787 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21206/statm: 16115 13362 93 168 0 15946 0
Current children cumulated CPU time (s) 402.3
Current children cumulated vsize (KiB) 64460

[startup+462.3 s]
/proc/loadavg: 1.99 1.97 1.98 3/178 21219
/proc/meminfo: memFree=17182644/32951124 swapFree=67102128/67111528
[pid=21206] ppid=21204 vsize=64460 CPUtime=462.3 cores=1,3,5,7
/proc/21206/stat : 21206 (march_hi) R 21204 21206 21168 0 -1 4202496 17017 0 0 0 46227 3 0 0 25 0 1 0 505435882 66007040 15233 33554432000 134512640 135197378 4294522576 18446744073709551615 134603252 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21206/statm: 16115 15233 93 168 0 15946 0
Current children cumulated CPU time (s) 462.3
Current children cumulated vsize (KiB) 64460

[startup+522.301 s]
/proc/loadavg: 1.99 1.97 1.98 3/178 21220
/proc/meminfo: memFree=17168616/32951124 swapFree=67102128/67111528
[pid=21206] ppid=21204 vsize=94736 CPUtime=522.3 cores=1,3,5,7
/proc/21206/stat : 21206 (march_hi) R 21204 21206 21168 0 -1 4202496 18926 0 0 0 52227 3 0 0 25 0 1 0 505435882 97009664 17142 33554432000 134512640 135197378 4294522576 18446744073709551615 134603172 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21206/statm: 23684 17142 93 168 0 23515 0
Current children cumulated CPU time (s) 522.3
Current children cumulated vsize (KiB) 94736

[startup+582.3 s]
/proc/loadavg: 1.99 1.97 1.98 3/178 21222
/proc/meminfo: memFree=17154696/32951124 swapFree=67102128/67111528
[pid=21206] ppid=21204 vsize=94736 CPUtime=582.3 cores=1,3,5,7
/proc/21206/stat : 21206 (march_hi) R 21204 21206 21168 0 -1 4202496 20792 0 0 0 58227 3 0 0 25 0 1 0 505435882 97009664 19008 33554432000 134512640 135197378 4294522576 18446744073709551615 134618141 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21206/statm: 23684 19008 93 168 0 23515 0
Current children cumulated CPU time (s) 582.3
Current children cumulated vsize (KiB) 94736

[startup+642.301 s]

################
# More data... #
################

[startup+4062.31 s]
/proc/loadavg: 2.00 2.00 2.00 3/178 21327
/proc/meminfo: memFree=16325112/32951124 swapFree=67102128/67111528
[pid=21206] ppid=21204 vsize=688932 CPUtime=4062.39 cores=1,3,5,7
/proc/21206/stat : 21206 (march_hi) R 21204 21206 21168 0 -1 4202496 127440 0 0 0 406213 26 0 0 25 0 1 0 505435882 705466368 125656 33554432000 134512640 135197378 4294522576 18446744073709551615 134603262 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21206/statm: 172233 125656 93 168 0 172064 0
Current children cumulated CPU time (s) 4062.39
Current children cumulated vsize (KiB) 688932

[startup+4122.3 s]
/proc/loadavg: 2.00 2.00 2.00 3/178 21328
/proc/meminfo: memFree=16311440/32951124 swapFree=67102128/67111528
[pid=21206] ppid=21204 vsize=688932 CPUtime=4122.39 cores=1,3,5,7
/proc/21206/stat : 21206 (march_hi) R 21204 21206 21168 0 -1 4202496 129300 0 0 0 412212 27 0 0 25 0 1 0 505435882 705466368 127516 33554432000 134512640 135197378 4294522576 18446744073709551615 134592702 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21206/statm: 172233 127516 93 168 0 172064 0
Current children cumulated CPU time (s) 4122.39
Current children cumulated vsize (KiB) 688932

[startup+4182.3 s]
/proc/loadavg: 2.00 2.00 2.00 3/178 21330
/proc/meminfo: memFree=16297284/32951124 swapFree=67102128/67111528
[pid=21206] ppid=21204 vsize=688932 CPUtime=4182.39 cores=1,3,5,7
/proc/21206/stat : 21206 (march_hi) R 21204 21206 21168 0 -1 4202496 131171 0 0 0 418212 27 0 0 25 0 1 0 505435882 705466368 129387 33554432000 134512640 135197378 4294522576 18446744073709551615 134524864 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21206/statm: 172233 129387 93 168 0 172064 0
Current children cumulated CPU time (s) 4182.39
Current children cumulated vsize (KiB) 688932

[startup+4242.3 s]
/proc/loadavg: 2.00 2.00 2.00 3/178 21332
/proc/meminfo: memFree=16283628/32951124 swapFree=67102128/67111528
[pid=21206] ppid=21204 vsize=688932 CPUtime=4242.39 cores=1,3,5,7
/proc/21206/stat : 21206 (march_hi) R 21204 21206 21168 0 -1 4202496 133022 0 0 0 424211 28 0 0 25 0 1 0 505435882 705466368 131238 33554432000 134512640 135197378 4294522576 18446744073709551615 134539114 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21206/statm: 172233 131238 93 168 0 172064 0
Current children cumulated CPU time (s) 4242.39
Current children cumulated vsize (KiB) 688932

[startup+4302.3 s]
/proc/loadavg: 2.00 2.00 2.00 3/178 21333
/proc/meminfo: memFree=16269204/32951124 swapFree=67102128/67111528
[pid=21206] ppid=21204 vsize=688932 CPUtime=4302.39 cores=1,3,5,7
/proc/21206/stat : 21206 (march_hi) R 21204 21206 21168 0 -1 4202496 134930 0 0 0 430211 28 0 0 25 0 1 0 505435882 705466368 133146 33554432000 134512640 135197378 4294522576 18446744073709551615 134603250 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21206/statm: 172233 133146 93 168 0 172064 0
Current children cumulated CPU time (s) 4302.39
Current children cumulated vsize (KiB) 688932

[startup+4362.3 s]
/proc/loadavg: 2.06 2.01 2.00 3/178 21335
/proc/meminfo: memFree=16255424/32951124 swapFree=67102128/67111528
[pid=21206] ppid=21204 vsize=688932 CPUtime=4362.4 cores=1,3,5,7
/proc/21206/stat : 21206 (march_hi) R 21204 21206 21168 0 -1 4202496 136783 0 0 0 436211 29 0 0 25 0 1 0 505435882 705466368 134999 33554432000 134512640 135197378 4294522576 18446744073709551615 134539117 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21206/statm: 172233 134999 93 168 0 172064 0
Current children cumulated CPU time (s) 4362.4
Current children cumulated vsize (KiB) 688932

[startup+4422.3 s]
/proc/loadavg: 2.02 2.01 2.00 3/178 21336
/proc/meminfo: memFree=16240772/32951124 swapFree=67102128/67111528
[pid=21206] ppid=21204 vsize=688932 CPUtime=4422.39 cores=1,3,5,7
/proc/21206/stat : 21206 (march_hi) R 21204 21206 21168 0 -1 4202496 138666 0 0 0 442210 29 0 0 25 0 1 0 505435882 705466368 136882 33554432000 134512640 135197378 4294522576 18446744073709551615 134550680 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21206/statm: 172233 136882 93 168 0 172064 0
Current children cumulated CPU time (s) 4422.39
Current children cumulated vsize (KiB) 688932

[startup+4482.3 s]
/proc/loadavg: 2.05 2.02 2.00 3/178 21338
/proc/meminfo: memFree=16226992/32951124 swapFree=67102128/67111528
[pid=21206] ppid=21204 vsize=688932 CPUtime=4482.39 cores=1,3,5,7
/proc/21206/stat : 21206 (march_hi) R 21204 21206 21168 0 -1 4202496 140550 0 0 0 448210 29 0 0 25 0 1 0 505435882 705466368 138766 33554432000 134512640 135197378 4294522576 18446744073709551615 134539102 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21206/statm: 172233 138766 93 168 0 172064 0
Current children cumulated CPU time (s) 4482.39
Current children cumulated vsize (KiB) 688932

[startup+4542.3 s]
/proc/loadavg: 2.02 2.01 2.00 3/178 21340
/proc/meminfo: memFree=16212692/32951124 swapFree=67102128/67111528
[pid=21206] ppid=21204 vsize=688932 CPUtime=4542.39 cores=1,3,5,7
/proc/21206/stat : 21206 (march_hi) R 21204 21206 21168 0 -1 4202496 142437 0 0 0 454210 29 0 0 25 0 1 0 505435882 705466368 140653 33554432000 134512640 135197378 4294522576 18446744073709551615 134524921 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21206/statm: 172233 140653 93 168 0 172064 0
Current children cumulated CPU time (s) 4542.39
Current children cumulated vsize (KiB) 688932

[startup+4602.3 s]
/proc/loadavg: 2.00 2.01 2.00 3/178 21341
/proc/meminfo: memFree=16197920/32951124 swapFree=67102128/67111528
[pid=21206] ppid=21204 vsize=688932 CPUtime=4602.4 cores=1,3,5,7
/proc/21206/stat : 21206 (march_hi) R 21204 21206 21168 0 -1 4202496 144277 0 0 0 460211 29 0 0 25 0 1 0 505435882 705466368 142493 33554432000 134512640 135197378 4294522576 18446744073709551615 134603283 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21206/statm: 172233 142493 93 168 0 172064 0
Current children cumulated CPU time (s) 4602.4
Current children cumulated vsize (KiB) 688932

[startup+4662.3 s]
/proc/loadavg: 2.00 2.00 2.00 3/178 21343
/proc/meminfo: memFree=16184012/32951124 swapFree=67102128/67111528
[pid=21206] ppid=21204 vsize=688932 CPUtime=4662.4 cores=1,3,5,7
/proc/21206/stat : 21206 (march_hi) R 21204 21206 21168 0 -1 4202496 146148 0 0 0 466210 30 0 0 25 0 1 0 505435882 705466368 144364 33554432000 134512640 135197378 4294522576 18446744073709551615 134548733 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21206/statm: 172233 144364 93 168 0 172064 0
Current children cumulated CPU time (s) 4662.4
Current children cumulated vsize (KiB) 688932

[startup+4722.31 s]
/proc/loadavg: 2.00 2.00 2.00 3/178 21344
/proc/meminfo: memFree=16169488/32951124 swapFree=67102128/67111528
[pid=21206] ppid=21204 vsize=688932 CPUtime=4722.41 cores=1,3,5,7
/proc/21206/stat : 21206 (march_hi) R 21204 21206 21168 0 -1 4202496 147991 0 0 0 472210 31 0 0 25 0 1 0 505435882 705466368 146207 33554432000 134512640 135197378 4294522576 18446744073709551615 134524895 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21206/statm: 172233 146207 93 168 0 172064 0
Current children cumulated CPU time (s) 4722.41
Current children cumulated vsize (KiB) 688932

[startup+4782.3 s]
/proc/loadavg: 2.00 2.00 2.00 3/178 21346
/proc/meminfo: memFree=16155328/32951124 swapFree=67102128/67111528
[pid=21206] ppid=21204 vsize=688932 CPUtime=4782.4 cores=1,3,5,7
/proc/21206/stat : 21206 (march_hi) R 21204 21206 21168 0 -1 4202496 149846 0 0 0 478209 31 0 0 25 0 1 0 505435882 705466368 148062 33554432000 134512640 135197378 4294522576 18446744073709551615 134603248 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21206/statm: 172233 148062 93 168 0 172064 0
Current children cumulated CPU time (s) 4782.4
Current children cumulated vsize (KiB) 688932

[startup+4842.3 s]
/proc/loadavg: 2.00 2.00 2.00 3/178 21357
/proc/meminfo: memFree=16139992/32951124 swapFree=67102128/67111528
[pid=21206] ppid=21204 vsize=688932 CPUtime=4842.4 cores=1,3,5,7
/proc/21206/stat : 21206 (march_hi) R 21204 21206 21168 0 -1 4202496 151643 0 0 0 484208 32 0 0 25 0 1 0 505435882 705466368 149859 33554432000 134512640 135197378 4294522576 18446744073709551615 134603252 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21206/statm: 172233 149859 93 168 0 172064 0
Current children cumulated CPU time (s) 4842.4
Current children cumulated vsize (KiB) 688932

[startup+4902.3 s]
/proc/loadavg: 2.08 2.02 2.01 3/178 21358
/proc/meminfo: memFree=16125948/32951124 swapFree=67102128/67111528
[pid=21206] ppid=21204 vsize=688932 CPUtime=4902.4 cores=1,3,5,7
/proc/21206/stat : 21206 (march_hi) R 21204 21206 21168 0 -1 4202496 153454 0 0 0 490208 32 0 0 25 0 1 0 505435882 705466368 151670 33554432000 134512640 135197378 4294522576 18446744073709551615 134603283 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21206/statm: 172233 151670 93 168 0 172064 0
Current children cumulated CPU time (s) 4902.4
Current children cumulated vsize (KiB) 688932

[startup+4962.3 s]
/proc/loadavg: 2.03 2.01 2.00 3/178 21360
/proc/meminfo: memFree=16112032/32951124 swapFree=67102128/67111528
[pid=21206] ppid=21204 vsize=688932 CPUtime=4962.4 cores=1,3,5,7
/proc/21206/stat : 21206 (march_hi) R 21204 21206 21168 0 -1 4202496 155260 0 0 0 496208 32 0 0 25 0 1 0 505435882 705466368 153476 33554432000 134512640 135197378 4294522576 18446744073709551615 134524895 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21206/statm: 172233 153476 93 168 0 172064 0
Current children cumulated CPU time (s) 4962.4
Current children cumulated vsize (KiB) 688932



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+4999.9 s]
/proc/loadavg: 2.09 2.02 2.01 2/176 21362
/proc/meminfo: memFree=16677716/32951124 swapFree=67102128/67111528
[pid=21206] ppid=21204 vsize=688932 CPUtime=5000.02 cores=1,3,5,7
/proc/21206/stat : 21206 (march_hi) R 21204 21206 21168 0 -1 4202496 156374 0 0 0 499969 33 0 0 25 0 1 0 505435882 705466368 154590 33554432000 134512640 135197378 4294522576 18446744073709551615 134603248 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21206/statm: 172233 154590 93 168 0 172064 0
Current children cumulated CPU time (s) 5000.02
Current children cumulated vsize (KiB) 688932

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+4999.9 s]
/proc/loadavg: 2.09 2.02 2.01 2/176 21362
/proc/meminfo: memFree=16677716/32951124 swapFree=67102128/67111528
[pid=21206] ppid=21204 vsize=688932 CPUtime=5000.02 cores=1,3,5,7
/proc/21206/stat : 21206 (march_hi) R 21204 21206 21168 0 -1 4202496 156374 0 0 0 499969 33 0 0 25 0 1 0 505435882 705466368 154590 33554432000 134512640 135197378 4294522576 18446744073709551615 134603248 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21206/statm: 172233 154590 93 168 0 172064 0
Current children cumulated CPU time (s) 5000.02
Current children cumulated vsize (KiB) 688932

Child ended because it received signal 15 (SIGTERM)
Real time (s): 4999.93
CPU time (s): 5000.05
CPU user time (s): 4999.69
CPU system time (s): 0.356945
CPU usage (%): 100.002
Max. virtual memory (cumulated for all children) (KiB): 688932

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4999.69
system time used= 0.356945
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 156374
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 3
involuntary context switches= 58063

runsolver used 7.10792 second user time and 15.6286 second system time

The end

Launcher Data 

Begin job on node119 at 2011-04-18 20:40:52
IDJOB=3210141
IDBENCH=82684
IDSOLVER=1627
FILE ID=node119/3210141-1303152052
RUNJOBID= node119-1303152052-21187
PBS_JOBID= 13020985
Free space on /tmp= 73044 MiB

SOLVER NAME= SAT09 reference solver: march_hi hi
BENCH NAME= SAT11/random/large/unif-k7-r85-v150-c12750-S1290198737-013-UNKNOWN.cnf
COMMAND LINE= HOME/march_hi BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3210141-1303152052/watcher-3210141-1303152052 -o /tmp/evaluation-result-3210141-1303152052/solver-3210141-1303152052 -C 5000 -W 5000 -M 15500  HOME/march_hi HOME/instance-3210141-1303152052.cnf

TIME LIMIT= 5000 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 707c4bcddcda116edc4a57a77dc3ba68
RANDOM SEED=2140512428

node119.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.63
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      17294532 kB
Buffers:        687788 kB
Cached:       14555860 kB
SwapCached:       5516 kB
Active:        4067652 kB
Inactive:     11240100 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      17294532 kB
SwapTotal:    67111528 kB
SwapFree:     67102128 kB
Dirty:            2848 kB
Writeback:           0 kB
AnonPages:       62532 kB
Mapped:          15364 kB
Slab:           283420 kB
PageTables:       5024 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   213924 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73044 MiB
End job on node119 at 2011-04-18 22:04:14