A basic concept in classical economics is that given certain plausible assumptions it is possible to define utility functions which measure how desirable economic actors find possible states of the world. Rational actors will then try to maximize the expected value of their utility function. For example most people will have an utility function which gives an additional two million dollars less than twice the value of an additional one million dollars. Hence they will prefer a sure million dollars to a 50% chance of two million dollars as this will maximize the expected value of their utility function.
The Capital Asset Pricing Model (CAPM) uses considerations of this sort to predict that risky assets will sell at a discount to their expected future value (as computed in dollars) and that the amount of the discount will increase as the amount of future uncertainty increases. As without such discounts investors would prefer to buy only the safest assets. It follows that risky investments will have greater expected return. Note risk here is referring to non-diversifiable risk. Risk particular to individual assets can be essentially eliminated by buying a diversified portfolio of such assets. However some risks (such that the economy as a whole will do badly) are not particular to individual assets and cannot be eliminated by diversification.
In the case of stocks it is reasonable to divide the risk (uncertainty in future returns) into two parts. That due to idiosyncratic factors particular to individual companies and that due to uncertainly about the general future trend of stock prices (as stocks tend to move up and down together). Stocks vary in how sensitive they are to general market movements. Some might tend to move up and down twice as much as the market, others only half as much as the market. The Greek letter beta is conventionally used to denote how sensitive the price of a particular individual stock is to a general change in the level of stock prices normalized so that a stock with a beta of x will tend to move up or down by x% when the general market moves up or down by 1%. High beta stocks will have more non-diversifiable risk and are predicted by the CAPM to have greater expected returns.
The CAPM is quite elegant mathematically. However that does not mean it is correct. Eric Falkenstein has extensively criticized it in books and his now dead blog, Falkenblog, which I mentioned earlier this month. Falkenstein's criticism (I don't know to what extent it is original, for the most part it is new to me) comes in two parts.
He claims that empirically high beta stocks have historically performed worse than low beta stocks which is a bit strange if their expected returns were actually higher. A big part of this seems to be due to the highest beta stocks performing badly with returns otherwise pretty flat with respect to beta.
On the theoretical side he points out the usual utility function framework is inadequate as it neglects the fact that people care about how they are doing relative to others. So they are going to prefer seeing their stocks go up 20% when the market is up 10% to seeing their stocks go up 20% when the market is up 30% although their personal return is the same in both cases. To the extent that people care more about relative returns than absolute returns (or as Falkenstein puts it are driven more by envy than by greed) the predictions of the CAPM will be flawed. For the so called risk free rate of return (often taken to be the interest rate paid on government bonds) is not actually risk free if people care (as they often will)about missing out on a big move upward by the stock market. The risk free investment for such people will be an index fund which guarantees them the average market return. Which means in effect that all risk is diversifiable and that there is no reason to anticipate greater expected returns when voluntarily assuming risk by deviating from the market average portfolio.