Trace number 1562656

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth-5-0 2009-03-19? (MO) 1198.92 607.472

General information on the benchmark

NameAPPLICATIONS/bioinfo/
q_query_3_l40_lambda.cnf
MD5SUM660e95de0ccce382abc30cb1546100bc
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark2.54361
Satisfiable
(Un)Satisfiability was proved
Number of variables28488
Number of clauses140544
Sum of the clauses size404432
Maximum clause length9
Minimum clause length1
Number of clauses of size 13800
Number of clauses of size 235941
Number of clauses of size 381078
Number of clauses of size 413589
Number of clauses of size 56016
Number of clauses of size over 5120

Solver Data 

0.28/0.39	c Command: HOME/ttsth-5-0 -num-threads 2 HOME/instance-1562656-1240997269.cnf 
0.28/0.39	c Using SATTIMEOUT of 1200
0.28/0.39	c Using SATRAM of 1800
0.28/0.39	c Size of subset node = 16
0.28/0.39	c Variables = 28488
0.28/0.39	c Clauses = 140544
0.28/0.39	c Literals = 404432
0.28/0.39	c Reduced size:
0.28/0.39	c Variables = 28488
0.28/0.39	c Clauses = 125907
0.28/0.39	c Literals = 367661
0.28/0.39	c a2
0.79/0.80	c a1
11.18/11.25	c Independent sub-problems = 1
11.18/11.25	c Sub-problem with 23780 variables and 125907 clauses
11.18/11.25	c using permutation method 4
11.18/11.25	c count = 1
11.18/11.25	c Ordering finished 11.180 seconds
11.18/11.25	c mapped 11.250 seconds
11.18/11.25	c permuted 11.250 seconds
11.18/11.25	c about to sort
11.28/11.30	c sorted
12.89/12.97	c clauses now 125907
12.89/12.97	c init finished 11.400 seconds
12.89/12.97	c tree built 12.970 seconds
12.89/12.97	c 
12.89/12.97	c PA progress                      | CPU progress Partial assignments | improvements      CPU time        PA/sec        nodes
12.89/12.97	c ---------------------------------|----------------------------------|------------------------------------------------------
12.89/12.97	c ................................ | ..........                       |                       13.0           0.0             
12.89/12.97	c ................................ | ..........                     1 |                       13.0           0.1             
12.89/12.97	c ****************................ | ..........                     2 |                       13.0           0.2            1
12.89/12.97	c ****************................ | ..........                     3 |                       13.0           0.2            1
12.89/12.97	c ****************................ | ..........                     4 |                       13.0           0.3            1
12.89/12.97	c ****************................ | ..........                     5 |                       13.0           0.4            1
12.89/12.97	c ****************................ | ..........                     6 |                       13.0           0.5            1
12.89/12.99	c ****************................ | ..........                  1000 | 2                     13.0          77.0          144
23.99/18.51	c ****************................ | ..........                177000 | 8991                  24.0        7371.9      1545858
47.94/30.58	c ****************................ | ..........                465000 | 24724                 48.1        9665.4      4224941
71.88/42.57	c ****************................ | ..........                637000 | 36569                 72.0        8843.5      7028988
96.03/54.66	c ****************................ | ..........                767000 | 44729                 96.2        7974.6      8954289
119.98/66.66	c ****************................ | *.........                850000 | 50939                120.1        7075.7     11509095
144.13/78.70	c ****************................ | *.........                953000 | 57124                144.2        6611.2     13476200
168.07/90.72	c ****************................ | *.........               1084000 | 64234                168.2        6446.6     14294345
191.82/102.68	c ****************................ | *.........               1267000 | 74215                192.0        6597.9     15654554
215.97/114.79	c ****************................ | *.........               1390000 | 81882                216.2        6429.2     17942572
240.12/126.89	c ****************................ | **........               1463000 | 87520                240.4        6086.7     20279062
264.27/138.92	c ****************................ | **........               1518000 | 92506                264.4        5742.2     21837754
288.41/151.03	c ****************................ | **........               1568000 | 97049                288.5        5434.6     23494397
311.97/162.86	c ****************................ | **........               1613000 | 101120               312.1        5167.6     25194200
335.92/174.89	c ****************................ | **........               1674000 | 105942               336.1        4980.1     27842491
360.08/186.94	c ****************................ | ***.......               1732000 | 110613               360.2        4808.3     30302718
383.81/198.87	c ****************................ | ***.......               1867000 | 118918               384.0        4861.9     32285508
407.96/210.90	c ****************................ | ***.......               2174000 | 137421               408.0        5328.0     34293703
431.92/222.93	c ****************................ | ***.......               2351000 | 148948               432.0        5441.6     37819687
456.05/235.01	c ****************................ | ***.......               2517000 | 159320               456.1        5517.9     40106588
479.80/246.95	c ****************................ | ****......               2651000 | 166915               480.0        5522.9     43460572
504.15/259.12	c ****************................ | ****......               2779000 | 174771               504.3        5510.9     46837877
527.91/271.09	c ****************................ | ****......               2913000 | 182459               528.2        5515.2     49861185
552.05/283.13	c ****************................ | ****......               3014000 | 189468               552.2        5458.1     53021488
575.80/295.07	c ****************................ | ****......               3153000 | 195831               576.0        5473.7     55238428
599.95/307.14	c ****************................ | *****.....               3324000 | 204519               600.1        5538.8     57584458
623.90/319.17	c ****************................ | *****.....               3458000 | 213052               624.1        5540.5     59657763
647.86/331.15	c ****************................ | *****.....               3600000 | 221435               648.1        5555.0     61137551
671.80/343.17	c ****************................ | *****.....               3718000 | 229647               672.0        5532.4     63550237
695.75/355.19	c ****************................ | *****.....               3865000 | 239260               696.0        5552.8     65248161
719.90/367.20	c ****************................ | ******....               3971000 | 246624               720.0        5515.1     67070130
743.84/379.26	c ****************................ | ******....               4075000 | 253836               744.1        5476.5     68939881
767.99/391.38	c ****************................ | ******....               4178000 | 261192               768.3        5438.2     70766904
791.74/403.29	c ****************................ | ******....               4271000 | 267727               792.0        5392.3     72318781
815.88/415.35	c ****************................ | ******....               4390000 | 274363               816.1        5379.2     74675606
839.84/427.33	c ****************................ | *******...               4497000 | 280858               840.0        5353.3     76785647
863.79/439.37	c ****************................ | *******...               4586000 | 287381               864.1        5307.5     79239354
887.73/451.38	c ****************................ | *******...               4679000 | 293316               888.0        5269.0     80874994
912.08/463.51	c ****************................ | *******...               4778000 | 299671               912.2        5237.6     82639515
935.82/475.42	c ****************................ | *******...               4877000 | 305742               936.0        5210.3     84559144
959.98/487.57	c ****************................ | ********..               4970000 | 311217               960.3        5175.7     86819449
983.74/499.48	c ****************................ | ********..               5049000 | 315788               984.0        5130.8     89237609
1007.68/511.49	c ****************................ | ********..               5140000 | 320994              1008.0        5099.1     91197988
1031.83/523.60	c ****************................ | ********..               5224000 | 326677              1032.2        5061.1     93273381
1055.98/535.68	c ****************................ | ********..               5292000 | 330997              1056.3        5009.9     95335994
1080.13/547.72	c ****************................ | *********.               5357000 | 335035              1080.3        4958.6     97261981
1103.67/559.58	c ****************................ | *********.               5416000 | 338945              1104.0        4905.8     98798688
1127.82/571.61	c ****************................ | *********.               5477000 | 342754              1128.0        4855.5    100273405
1151.93/583.75	c ****************................ | *********.               5575000 | 348287              1152.2        4838.5    102550660
1175.76/595.73	c ****************................ | *********.               5644000 | 353041              1176.0        4799.2    105087830

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver version 3.2.9 (svn:492) (c) roussel@cril.univ-artois.fr

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-1562656-1240997269/watcher-1562656-1240997269 -o /tmp/evaluation-result-1562656-1240997269/solver-1562656-1240997269 -C 1200 -W 1800 -M 1800 HOME/ttsth-5-0 -num-threads 2 HOME/instance-1562656-1240997269.cnf 

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1200 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1230 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 1843200 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 1894400 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 0.00 0.00 0.00 3/65 6980
/proc/meminfo: memFree=1672824/2055920 swapFree=4175168/4192956
[pid=6980] ppid=6978 vsize=1064 CPUtime=0
/proc/6980/stat : 6980 (ttsth-5-0) R 6978 6980 6943 0 -1 4194304 149 0 0 0 0 0 0 0 19 0 1 0 127756907 1089536 133 1992294400 134512640 135237372 4294956192 18446744073709551615 134732614 0 0 4096 0 0 0 0 17 1 0 0
/proc/6980/statm: 266 134 77 176 0 88 0

[startup+0.0490391 s]
/proc/loadavg: 0.00 0.00 0.00 3/65 6980
/proc/meminfo: memFree=1672824/2055920 swapFree=4175168/4192956
[pid=6980] ppid=6978 vsize=2708 CPUtime=0.04
/proc/6980/stat : 6980 (ttsth-5-0) R 6978 6980 6943 0 -1 4194304 562 0 0 0 4 0 0 0 19 0 1 0 127756907 2772992 546 1992294400 134512640 135237372 4294956192 18446744073709551615 134639534 0 0 4096 0 0 0 0 17 1 0 0
/proc/6980/statm: 677 546 78 176 0 499 0
Current children cumulated CPU time (s) 0.04
Current children cumulated vsize (KiB) 2708

[startup+0.101045 s]
/proc/loadavg: 0.00 0.00 0.00 3/65 6980
/proc/meminfo: memFree=1672824/2055920 swapFree=4175168/4192956
[pid=6980] ppid=6978 vsize=4636 CPUtime=0.09
/proc/6980/stat : 6980 (ttsth-5-0) R 6978 6980 6943 0 -1 4194304 1035 0 0 0 9 0 0 0 19 0 1 0 127756907 4747264 1019 1992294400 134512640 135237372 4294956192 18446744073709551615 134639534 0 0 4096 0 0 0 0 17 1 0 0
/proc/6980/statm: 1159 1019 78 176 0 981 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 4636

[startup+0.301065 s]
/proc/loadavg: 0.00 0.00 0.00 3/65 6980
/proc/meminfo: memFree=1672824/2055920 swapFree=4175168/4192956
[pid=6980] ppid=6978 vsize=10744 CPUtime=0.28
/proc/6980/stat : 6980 (ttsth-5-0) R 6978 6980 6943 0 -1 4194304 2468 0 0 0 28 0 0 0 21 0 1 0 127756907 11001856 2452 1992294400 134512640 135237372 4294956192 18446744073709551615 134534101 0 0 4096 0 0 0 0 17 1 0 0
/proc/6980/statm: 2686 2452 78 176 0 2508 0
Current children cumulated CPU time (s) 0.28
Current children cumulated vsize (KiB) 10744

[startup+0.701107 s]
/proc/loadavg: 0.00 0.00 0.00 3/65 6980
/proc/meminfo: memFree=1672824/2055920 swapFree=4175168/4192956
[pid=6980] ppid=6978 vsize=20236 CPUtime=0.69
/proc/6980/stat : 6980 (ttsth-5-0) R 6978 6980 6943 0 -1 4194304 4771 0 0 0 68 1 0 0 24 0 1 0 127756907 20721664 4754 1992294400 134512640 135237372 4294956192 18446744073709551615 134519991 0 0 4096 0 0 0 0 17 1 0 0
/proc/6980/statm: 5059 4754 79 176 0 4881 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 20236

[startup+1.50119 s]
/proc/loadavg: 0.00 0.00 0.00 2/66 6981
/proc/meminfo: memFree=1654064/2055920 swapFree=4175168/4192956
[pid=6980] ppid=6978 vsize=23928 CPUtime=1.49
/proc/6980/stat : 6980 (ttsth-5-0) R 6978 6980 6943 0 -1 4194304 6209 0 0 0 148 1 0 0 25 0 1 0 127756907 24502272 5810 1992294400 134512640 135237372 4294956192 18446744073709551615 134600966 0 0 4096 0 0 0 0 17 1 0 0
/proc/6980/statm: 5982 5810 80 176 0 5804 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 23928

[startup+3.10236 s]
/proc/loadavg: 0.00 0.00 0.00 2/66 6981
/proc/meminfo: memFree=1649712/2055920 swapFree=4175168/4192956
[pid=6980] ppid=6978 vsize=24112 CPUtime=3.09
/proc/6980/stat : 6980 (ttsth-5-0) R 6978 6980 6943 0 -1 4194304 6226 0 0 0 308 1 0 0 25 0 1 0 127756907 24690688 5827 1992294400 134512640 135237372 4294956192 18446744073709551615 134600972 0 0 4096 0 0 0 0 17 1 0 0
/proc/6980/statm: 6028 5827 80 176 0 5850 0
Current children cumulated CPU time (s) 3.09
Current children cumulated vsize (KiB) 24112

[startup+6.3017 s]
/proc/loadavg: 0.08 0.02 0.01 2/66 6981
/proc/meminfo: memFree=1649392/2055920 swapFree=4175168/4192956
[pid=6980] ppid=6978 vsize=24244 CPUtime=6.28
/proc/6980/stat : 6980 (ttsth-5-0) R 6978 6980 6943 0 -1 4194304 6282 0 0 0 627 1 0 0 25 0 1 0 127756907 24825856 5883 1992294400 134512640 135237372 4294956192 18446744073709551615 134600718 0 0 4096 0 0 0 0 17 1 0 0
/proc/6980/statm: 6061 5883 80 176 0 5883 0
Current children cumulated CPU time (s) 6.28
Current children cumulated vsize (KiB) 24244

[startup+12.7014 s]
/proc/loadavg: 0.15 0.03 0.01 2/66 6981
/proc/meminfo: memFree=1620336/2055920 swapFree=4175168/4192956
[pid=6980] ppid=6978 vsize=78628 CPUtime=12.69
/proc/6980/stat : 6980 (ttsth-5-0) R 6978 6980 6943 0 -1 4194304 20133 0 0 0 1262 7 0 0 25 0 1 0 127756907 80515072 19488 1992294400 134512640 135237372 4294956192 18446744073709551615 134551487 0 0 4096 0 0 0 0 17 1 0 0
/proc/6980/statm: 19657 19488 94 176 0 19479 0
Current children cumulated CPU time (s) 12.69
Current children cumulated vsize (KiB) 78628

[startup+25.5017 s]
/proc/loadavg: 0.49 0.11 0.04 3/68 6983
/proc/meminfo: memFree=1541664/2055920 swapFree=4175168/4192956
[pid=6980] ppid=6978 vsize=154276 CPUtime=37.95
/proc/6980/stat : 6980 (ttsth-5-0) S 6978 6980 6943 0 -1 4194304 33773 0 0 0 3780 15 0 0 25 0 3 0 127756907 157978624 33128 1992294400 134512640 135237372 4294956192 18446744073709551615 134629544 0 0 4096 0 18446744073709551615 0 0 17 1 0 0
/proc/6980/statm: 38569 33128 98 176 0 38391 0
[pid=6980/tid=6982] ppid=6978 vsize=154276 CPUtime=12.51
/proc/6980/task/6982/stat : 6982 (ttsth-5-0) R 6978 6980 6943 0 -1 4194368 6076 0 0 0 1248 3 0 0 25 0 3 0 127758205 157978624 33128 1992294400 134512640 135237372 4294956192 18446744073709551615 134519949 0 0 4096 0 0 0 0 -1 1 0 0
[pid=6980/tid=6983] ppid=6978 vsize=154276 CPUtime=12.46
/proc/6980/task/6983/stat : 6983 (ttsth-5-0) R 6978 6980 6943 0 -1 4194368 7190 0 0 0 1242 4 0 0 25 0 3 0 127758205 157978624 33128 1992294400 134512640 135237372 4294956192 18446744073709551615 134591514 0 0 4096 0 0 0 0 -1 0 0 0
Current children cumulated CPU time (s) 37.95
Current children cumulated vsize (KiB) 154276

[startup+51.1015 s]
/proc/loadavg: 1.01 0.26 0.09 3/68 6983
/proc/meminfo: memFree=1458592/2055920 swapFree=4175168/4192956
[pid=6980] ppid=6978 vsize=239792 CPUtime=89.05
/proc/6980/stat : 6980 (ttsth-5-0) S 6978 6980 6943 0 -1 4194304 54619 0 0 0 8879 26 0 0 25 0 3 0 127756907 245547008 53974 1992294400 134512640 135237372 4294956192 18446744073709551615 134629544 0 0 4096 0 18446744073709551615 0 0 17 1 0 0
/proc/6980/statm: 59948 53974 98 176 0 59770 0
[pid=6980/tid=6982] ppid=6978 vsize=239792 CPUtime=38.1
/proc/6980/task/6982/stat : 6982 (ttsth-5-0) R 6978 6980 6943 0 -1 4194368 15080 0 0 0 3803 7 0 0 25 0 3 0 127758205 245547008 53974 1992294400 134512640 135237372 4294956192 18446744073709551615 134680727 0 0 4096 0 0 0 0 -1 1 0 0
[pid=6980/tid=6983] ppid=6978 vsize=239792 CPUtime=37.96
/proc/6980/task/6983/stat : 6983 (ttsth-5-0) R 6978 6980 6943 0 -1 4194368 19032 0 0 0 3786 10 0 0 25 0 3 0 127758205 245547008 53974 1992294400 134512640 135237372 4294956192 18446744073709551615 134591082 0 0 4096 0 0 0 0 -1 0 0 0
Current children cumulated CPU time (s) 89.05
Current children cumulated vsize (KiB) 239792

[startup+102.306 s]
/proc/loadavg: 1.57 0.53 0.18 3/68 6983
/proc/meminfo: memFree=1343264/2055920 swapFree=4175168/4192956
[pid=6980] ppid=6978 vsize=355348 CPUtime=191.23
/proc/6980/stat : 6980 (ttsth-5-0) S 6978 6980 6943 0 -1 4194304 82944 0 0 0 19079 44 0 0 25 0 3 0 127756907 363876352 82299 1992294400 134512640 135237372 4294956192 18446744073709551615 134629544 0 0 4096 0 18446744073709551615 0 0 17 1 0 0
/proc/6980/statm: 88837 82299 98 176 0 88659 0
[pid=6980/tid=6982] ppid=6978 vsize=355348 CPUtime=89.29
/proc/6980/task/6982/stat : 6982 (ttsth-5-0) R 6978 6980 6943 0 -1 4194368 26117 0 0 0 8915 14 0 0 25 0 3 0 127758205 363876352 82299 1992294400 134512640 135237372 4294956192 18446744073709551615 134562272 0 0 4096 0 0 0 0 -1 1 0 0
[pid=6980/tid=6983] ppid=6978 vsize=355348 CPUtime=88.95
/proc/6980/task/6983/stat : 6983 (ttsth-5-0) R 6978 6980 6943 0 -1 4194368 36320 0 0 0 8874 21 0 0 25 0 3 0 127758205 363876352 82299 1992294400 134512640 135237372 4294956192 18446744073709551615 134749351 0 0 4096 0 0 0 0 -1 0 0 0
Current children cumulated CPU time (s) 191.23
Current children cumulated vsize (KiB) 355348

[startup+162.301 s]
/proc/loadavg: 1.84 0.79 0.30 3/68 6983
/proc/meminfo: memFree=1194592/2055920 swapFree=4175168/4192956
[pid=6980] ppid=6978 vsize=513780 CPUtime=310.97
/proc/6980/stat : 6980 (ttsth-5-0) S 6978 6980 6943 0 -1 4194304 120237 0 0 0 31032 65 0 0 25 0 3 0 127756907 526110720 119592 1992294400 134512640 135237372 4294956192 18446744073709551615 134629544 0 0 4096 0 18446744073709551615 0 0 17 1 0 0
/proc/6980/statm: 128445 119592 98 176 0 128267 0
[pid=6980/tid=6982] ppid=6978 vsize=513780 CPUtime=149.27
/proc/6980/task/6982/stat : 6982 (ttsth-5-0) R 6978 6980 6943 0 -1 4194368 39694 0 0 0 14903 24 0 0 25 0 3 0 127758205 526110720 119592 1992294400 134512640 135237372 4294956192 18446744073709551615 134562437 0 0 4096 0 0 0 0 -1 1 0 0
[pid=6980/tid=6983] ppid=6978 vsize=513780 CPUtime=148.71
/proc/6980/task/6983/stat : 6983 (ttsth-5-0) R 6978 6980 6943 0 -1 4194368 60036 0 0 0 14838 33 0 0 25 0 3 0 127758205 526110720 119592 1992294400 134512640 135237372 4294956192 18446744073709551615 134680499 0 0 4096 0 0 0 0 -1 0 0 0
Current children cumulated CPU time (s) 310.97
Current children cumulated vsize (KiB) 513780

[startup+222.302 s]
/proc/loadavg: 1.94 1.01 0.40 3/68 6983
/proc/meminfo: memFree=997344/2055920 swapFree=4175168/4192956
[pid=6980] ppid=6978 vsize=715172 CPUtime=430.71
/proc/6980/stat : 6980 (ttsth-5-0) S 6978 6980 6943 0 -1 4194304 169545 0 0 0 42983 88 0 0 25 0 3 0 127756907 732336128 168900 1992294400 134512640 135237372 4294956192 18446744073709551615 134629544 0 0 4096 0 18446744073709551615 0 0 17 1 0 0
/proc/6980/statm: 178793 168900 98 176 0 178615 0
[pid=6980/tid=6982] ppid=6978 vsize=715172 CPUtime=209.26
/proc/6980/task/6982/stat : 6982 (ttsth-5-0) R 6978 6980 6943 0 -1 4194368 61400 0 0 0 20891 35 0 0 25 0 3 0 127758205 732336128 168900 1992294400 134512640 135237372 4294956192 18446744073709551615 134562338 0 0 4096 0 0 0 0 -1 1 0 0
[pid=6980/tid=6983] ppid=6978 vsize=715172 CPUtime=208.47
/proc/6980/task/6983/stat : 6983 (ttsth-5-0) R 6978 6980 6943 0 -1 4194368 87638 0 0 0 20802 45 0 0 25 0 3 0 127758205 732336128 168900 1992294400 134512640 135237372 4294956192 18446744073709551615 134562341 0 0 4096 0 0 0 0 -1 0 0 0
Current children cumulated CPU time (s) 430.71
Current children cumulated vsize (KiB) 715172

[startup+282.302 s]
/proc/loadavg: 1.98 1.19 0.50 3/68 6983
/proc/meminfo: memFree=761696/2055920 swapFree=4175168/4192956
[pid=6980] ppid=6978 vsize=961312 CPUtime=550.45
/proc/6980/stat : 6980 (ttsth-5-0) S 6978 6980 6943 0 -1 4194304 228798 0 0 0 54926 119 0 0 25 0 3 0 127756907 984383488 228153 1992294400 134512640 135237372 4294956192 18446744073709551615 134629544 0 0 4096 0 18446744073709551615 0 0 17 1 0 0
/proc/6980/statm: 240328 228153 98 176 0 240150 0
[pid=6980/tid=6982] ppid=6978 vsize=961312 CPUtime=269.24
/proc/6980/task/6982/stat : 6982 (ttsth-5-0) R 6978 6980 6943 0 -1 4194368 101226 0 0 0 26866 58 0 0 25 0 3 0 127758205 984383488 228153 1992294400 134512640 135237372 4294956192 18446744073709551615 134562346 0 0 4096 0 0 0 0 -1 1 0 0
[pid=6980/tid=6983] ppid=6978 vsize=961312 CPUtime=268.24
/proc/6980/task/6983/stat : 6983 (ttsth-5-0) R 6978 6980 6943 0 -1 4194368 107065 0 0 0 26770 54 0 0 25 0 3 0 127758205 984383488 228153 1992294400 134512640 135237372 4294956192 18446744073709551615 134587914 0 0 4096 0 0 0 0 -1 0 0 0
Current children cumulated CPU time (s) 550.45
Current children cumulated vsize (KiB) 961312

[startup+342.301 s]
/proc/loadavg: 1.99 1.33 0.59 3/68 6983
/proc/meminfo: memFree=592288/2055920 swapFree=4175168/4192956
[pid=6980] ppid=6978 vsize=1131300 CPUtime=670.19
/proc/6980/stat : 6980 (ttsth-5-0) S 6978 6980 6943 0 -1 4194304 270407 0 0 0 66876 143 0 0 25 0 3 0 127756907 1158451200 269762 1992294400 134512640 135237372 4294956192 18446744073709551615 134629544 0 0 4096 0 18446744073709551615 0 0 17 1 0 0
/proc/6980/statm: 282825 269762 98 176 0 282647 0
[pid=6980/tid=6982] ppid=6978 vsize=1131300 CPUtime=329.22
/proc/6980/task/6982/stat : 6982 (ttsth-5-0) R 6978 6980 6943 0 -1 4194368 131157 0 0 0 32848 74 0 0 25 0 3 0 127758205 1158451200 269762 1992294400 134512640 135237372 4294956192 18446744073709551615 134749351 0 0 4096 0 0 0 0 -1 1 0 0
[pid=6980/tid=6983] ppid=6978 vsize=1131300 CPUtime=327.99
/proc/6980/task/6983/stat : 6983 (ttsth-5-0) R 6978 6980 6943 0 -1 4194368 118743 0 0 0 32738 61 0 0 25 0 3 0 127758205 1158451200 269762 1992294400 134512640 135237372 4294956192 18446744073709551615 134587921 0 0 4096 0 0 0 0 -1 0 0 0
Current children cumulated CPU time (s) 670.19
Current children cumulated vsize (KiB) 1131300

[startup+402.302 s]
/proc/loadavg: 1.99 1.45 0.67 3/68 6983
/proc/meminfo: memFree=454816/2055920 swapFree=4175168/4192956
[pid=6980] ppid=6978 vsize=1274012 CPUtime=789.95
/proc/6980/stat : 6980 (ttsth-5-0) S 6978 6980 6943 0 -1 4194304 304832 0 0 0 78832 163 0 0 25 0 3 0 127756907 1304588288 304187 1992294400 134512640 135237372 4294956192 18446744073709551615 134629544 0 0 4096 0 18446744073709551615 0 0 17 1 0 0
/proc/6980/statm: 318503 304187 98 176 0 318325 0
[pid=6980/tid=6982] ppid=6978 vsize=1274012 CPUtime=389.21
/proc/6980/task/6982/stat : 6982 (ttsth-5-0) R 6978 6980 6943 0 -1 4194368 153258 0 0 0 38833 88 0 0 25 0 3 0 127758205 1304588288 304187 1992294400 134512640 135237372 4294956192 18446744073709551615 134562351 0 0 4096 0 0 0 0 -1 1 0 0
[pid=6980/tid=6983] ppid=6978 vsize=1274012 CPUtime=387.75
/proc/6980/task/6983/stat : 6983 (ttsth-5-0) R 6978 6980 6943 0 -1 4194368 131067 0 0 0 38708 67 0 0 25 0 3 0 127758205 1304588288 304187 1992294400 134512640 135237372 4294956192 18446744073709551615 134562459 0 0 4096 0 0 0 0 -1 0 0 0
Current children cumulated CPU time (s) 789.95
Current children cumulated vsize (KiB) 1274012

[startup+462.301 s]
/proc/loadavg: 1.99 1.55 0.75 3/68 6983
/proc/meminfo: memFree=293536/2055920 swapFree=4175168/4192956
[pid=6980] ppid=6978 vsize=1439920 CPUtime=909.68
/proc/6980/stat : 6980 (ttsth-5-0) S 6978 6980 6943 0 -1 4194304 344894 0 0 0 90785 183 0 0 25 0 3 0 127756907 1474478080 344249 1992294400 134512640 135237372 4294956192 18446744073709551615 134629544 0 0 4096 0 18446744073709551615 0 0 17 1 0 0
/proc/6980/statm: 359980 344249 98 176 0 359802 0
[pid=6980/tid=6982] ppid=6978 vsize=1439920 CPUtime=449.19
/proc/6980/task/6982/stat : 6982 (ttsth-5-0) R 6978 6980 6943 0 -1 4194368 173739 0 0 0 44820 99 0 0 25 0 3 0 127758205 1474478080 344249 1992294400 134512640 135237372 4294956192 18446744073709551615 134562338 0 0 4096 0 0 0 0 -1 1 0 0
[pid=6980/tid=6983] ppid=6978 vsize=1439920 CPUtime=447.5
/proc/6980/task/6983/stat : 6983 (ttsth-5-0) R 6978 6980 6943 0 -1 4194368 150648 0 0 0 44674 76 0 0 25 0 3 0 127758205 1474478080 344249 1992294400 134512640 135237372 4294956192 18446744073709551615 134549922 0 0 4096 0 0 0 0 -1 0 0 0
Current children cumulated CPU time (s) 909.68
Current children cumulated vsize (KiB) 1439920

[startup+522.301 s]
/proc/loadavg: 2.05 1.64 0.83 3/68 6983
/proc/meminfo: memFree=130528/2055920 swapFree=4175168/4192956
[pid=6980] ppid=6978 vsize=1611876 CPUtime=1029.43
/proc/6980/stat : 6980 (ttsth-5-0) S 6978 6980 6943 0 -1 4194304 386160 0 0 0 102737 206 0 0 25 0 3 0 127756907 1650561024 385515 1992294400 134512640 135237372 4294956192 18446744073709551615 134629544 0 0 4096 0 18446744073709551615 0 0 17 1 0 0
/proc/6980/statm: 402969 385515 98 176 0 402791 0
[pid=6980/tid=6982] ppid=6978 vsize=1611876 CPUtime=509.18
/proc/6980/task/6982/stat : 6982 (ttsth-5-0) R 6978 6980 6943 0 -1 4194368 195647 0 0 0 50806 112 0 0 25 0 3 0 127758205 1650561024 385515 1992294400 134512640 135237372 4294956192 18446744073709551615 134562278 0 0 4096 0 0 0 0 -1 1 0 0
[pid=6980/tid=6983] ppid=6978 vsize=1611876 CPUtime=507.27
/proc/6980/task/6983/stat : 6983 (ttsth-5-0) R 6978 6980 6943 0 -1 4194368 170006 0 0 0 50641 86 0 0 25 0 3 0 127758205 1650561024 385515 1992294400 134512640 135237372 4294956192 18446744073709551615 134562338 0 0 4096 0 0 0 0 -1 0 0 0
Current children cumulated CPU time (s) 1029.43
Current children cumulated vsize (KiB) 1611876

[startup+582.302 s]
/proc/loadavg: 2.02 1.71 0.90 3/68 6983
/proc/meminfo: memFree=16224/2055920 swapFree=4175168/4192956
[pid=6980] ppid=6978 vsize=1763248 CPUtime=1149.15
/proc/6980/stat : 6980 (ttsth-5-0) S 6978 6980 6943 0 -1 4194304 422393 0 0 0 114690 225 0 0 25 0 3 0 127756907 1805565952 421748 1992294400 134512640 135237372 4294956192 18446744073709551615 134629544 0 0 4096 0 18446744073709551615 0 0 17 1 0 0
/proc/6980/statm: 440812 421748 98 176 0 440634 0
[pid=6980/tid=6982] ppid=6978 vsize=1763248 CPUtime=569.15
/proc/6980/task/6982/stat : 6982 (ttsth-5-0) R 6978 6980 6943 0 -1 4194368 215006 0 0 0 56794 121 0 0 25 0 3 0 127758205 1805565952 421748 1992294400 134512640 135237372 4294956192 18446744073709551615 134749335 0 0 4096 0 0 0 0 -1 1 0 0
[pid=6980/tid=6983] ppid=6978 vsize=1763248 CPUtime=567.01
/proc/6980/task/6983/stat : 6983 (ttsth-5-0) R 6978 6980 6943 0 -1 4194368 186880 0 0 0 56605 96 0 0 25 0 3 0 127758205 1805565952 421749 1992294400 134512640 135237372 4294956192 18446744073709551615 134562444 0 0 4096 0 0 0 0 -1 0 0 0
Current children cumulated CPU time (s) 1149.15
Current children cumulated vsize (KiB) 1763248



Maximum VSize exceeded: sending SIGTERM then SIGKILL

[startup+607.206 s]
/proc/loadavg: 2.01 1.73 0.93 3/68 6983
/proc/meminfo: memFree=15456/2055920 swapFree=4175168/4192956
[pid=6980] ppid=6978 vsize=1844100 CPUtime=1198.64
/proc/6980/stat : 6980 (ttsth-5-0) S 6978 6980 6943 0 -1 4194304 441857 0 0 0 119627 237 0 0 25 0 3 0 127756907 1888358400 441212 1992294400 134512640 135237372 4294956192 18446744073709551615 134629544 0 0 4096 0 18446744073709551615 0 0 17 1 0 0
/proc/6980/statm: 461025 441212 98 176 0 460847 0
[pid=6980/tid=6982] ppid=6978 vsize=1844100 CPUtime=593.96
/proc/6980/task/6982/stat : 6982 (ttsth-5-0) R 6978 6980 6943 0 -1 4194368 223488 0 0 0 59269 127 0 0 25 0 3 0 127758205 1888358400 441212 1992294400 134512640 135237372 4294956192 18446744073709551615 134732535 0 0 4096 0 0 0 0 -1 1 0 0
[pid=6980/tid=6983] ppid=6978 vsize=1844100 CPUtime=591.68
/proc/6980/task/6983/stat : 6983 (ttsth-5-0) R 6978 6980 6943 0 -1 4194368 197862 0 0 0 59067 101 0 0 25 0 3 0 127758205 1888358400 441212 1992294400 134512640 135237372 4294956192 18446744073709551615 134562328 0 0 4096 0 0 0 0 -1 0 0 0
Current children cumulated CPU time (s) 1198.64
Current children cumulated vsize (KiB) 1844100

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+607.403 s]
/proc/loadavg: 2.01 1.73 0.93 3/68 6983
/proc/meminfo: memFree=15456/2055920 swapFree=4175168/4192956
[pid=6980] ppid=6978 vsize=0 CPUtime=1198.84
/proc/6980/stat : 6980 (ttsth-5-0) Z 6978 6980 6943 0 -1 4195340 441866 0 0 0 119628 256 0 0 16 0 2 0 127756907 0 0 1992294400 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0
/proc/6980/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 1198.84
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 607.472
CPU time (s): 1198.92
CPU user time (s): 1196.29
CPU system time (s): 2.6296
CPU usage (%): 197.362
Max. virtual memory (cumulated for all children) (KiB): 1844100

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1196.29
system time used= 2.6296
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 441866
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 102
involuntary context switches= 9162

runsolver used 0.601908 second user time and 1.62475 second system time

The end

Launcher Data 

Begin job on node81 at 2009-04-29 11:27:49
IDJOB=1562656
IDBENCH=70938
IDSOLVER=549
FILE ID=node81/1562656-1240997269
PBS_JOBID= 9204791
Free space on /tmp= 126608 MiB

SOLVER NAME= ttsth-5-0 2009-03-19
BENCH NAME= SAT09/APPLICATIONS/bioinfo/q_query_3_l40_lambda.cnf
COMMAND LINE= HOME/ttsth-5-0 -num-threads MAXNBTHREAD BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-1562656-1240997269/watcher-1562656-1240997269 -o /tmp/evaluation-result-1562656-1240997269/solver-1562656-1240997269 -C 1200 -W 1800 -M 1800  HOME/ttsth-5-0 -num-threads 2 HOME/instance-1562656-1240997269.cnf

TIME LIMIT= 1200 seconds
MEMORY LIMIT= 1800 MiB
MAX NB THREAD= 2

MD5SUM BENCH= 660e95de0ccce382abc30cb1546100bc
RANDOM SEED=591743165

node81.alineos.net Linux 2.6.9-22.EL.rootsmp #1 SMP Mon Oct 3 08:59:52 CEST 2005

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 15
model		: 4
model name	:                   Intel(R) Xeon(TM) CPU 3.00GHz
stepping	: 3
cpu MHz		: 3000.234
cache size	: 2048 KB
fpu		: yes
fpu_exception	: yes
cpuid level	: 5
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor ds_cpl cid cx16 xtpr
bogomips	: 5914.62
clflush size	: 64
cache_alignment	: 128
address sizes	: 36 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 15
model		: 4
model name	:                   Intel(R) Xeon(TM) CPU 3.00GHz
stepping	: 3
cpu MHz		: 3000.234
cache size	: 2048 KB
fpu		: yes
fpu_exception	: yes
cpuid level	: 5
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor ds_cpl cid cx16 xtpr
bogomips	: 5996.54
clflush size	: 64
cache_alignment	: 128
address sizes	: 36 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:      2055920 kB
MemFree:       1673176 kB
Buffers:         42236 kB
Cached:         284024 kB
SwapCached:      11360 kB
Active:         133060 kB
Inactive:       205876 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:      2055920 kB
LowFree:       1673176 kB
SwapTotal:     4192956 kB
SwapFree:      4175168 kB
Dirty:            3584 kB
Writeback:           0 kB
Mapped:          13568 kB
Slab:            29452 kB
Committed_AS:   506440 kB
PageTables:       1452 kB
VmallocTotal: 536870911 kB
VmallocUsed:    264960 kB
VmallocChunk: 536605679 kB
HugePages_Total:     0
HugePages_Free:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 126608 MiB
End job on node81 at 2009-04-29 11:38:00