There is no single index that is used to determine mortgage rates and, in the case of fixed-rate mortgage loans, no index is used.  To find out how fixed-rate mortgages are determined, I direct your attention to the MBS section of MND and suggest that you read up on the the edifying and informational contributions made in that section.

For adjustable-rate mortgages, some of the the most common indices are 1) the prime rate, 2) LIBOR and 3) the 1-year Treasury Bill.  The prime rate is most often used as an index for HELOC's.  The 1-year Treasury Bill was traditionally the index used for 1,3,5 and 7 year conventional ARM's for a long time.  At one point, LIBOR was also used for conventional ARM's, but its use faded once it started to go up.  Most conventional lenders have reverted to using the 1 year Treasury.  LIBOR was also the most commonly used index used to determine subprime mortgages.  Early on, Option Arms were tied to things like the Cost of Savings Index and the Cost of Funds Index and later to LIBOR or the 1 Year Constant Maturity Treasury Index (which is the average yield on United States Treasury securities adjusted to a constant maturity of 1 year). 

Good luck!