I recently reread "A Random Walk Down Wall Street" by Burton G. Malkiel. I had read an early edition (perhaps the first which came out in 1973) of this book long ago and liked it quite a bit so when I saw the "revised and updated" 10th edition ((c) 2011) in my local library I checked it out. I was a bit disappointed. While some material has been added the basic themes are the same and the ideas which were new and interesting to me when I first encountered them were considerably less so some thirty years later.
Which is not to say these ideas are wrong, I think the book has held up pretty well in that respect. Malkiel's main argument is that it is generally a bad idea to pay people to pick stocks for you because there is little evidence that there are professional stock pickers who consistently do any better than chance, throwing darts at the stock page in the newspaper being the traditional analogy. Which means they will tend to lag the market after you deduct their fees. There was and is considerable evidence that this is true. It isn't too surprising that the average performance of professionals is just average because professionals dominate the market so their performance as a group will not vary much from the performance of the market as a whole. However over any given time period some professionals will do well and others poorly. So perhaps we should just be sure to pick a manager who has done well in the past. Unfortunately (and a bit more surprising) the performance (good or bad) of portfolio managers shows little consistency over time. A manager might perform relatively well over a 5 or 10 year period and then relatively poorly over the next 5 or 10 year period. In fact the evidence is quite consistent with variations in performance being primarily due to luck and not differences in skill. This makes Malkiel's recommendation, low cost index funds, an attractive alternative to actively managed funds. As Malkiel points out in addition to their lower management fees index funds will also incur lower trading costs and will tend to be more tax efficient. And in fact index funds have grown greatly in popularity since the first edition of this book.
So why is it so hard to beat the market? One obvious possibility is that stock market prices being the weighted opinion of many intelligent and knowledgeable people are largely "correct" reflecting everything that is publicly known about a company's prospects. This is the efficient markets hypothesis (EMH). The EMH gets ridiculed a lot as the market's collective judgement sometimes seems quite far off the mark as with the bubble in internet stocks. However the EMH still seems like a reasonable explanation (even if it is not entirely correct) for why it is difficult to beat the market consistently as it is lot harder to identify market irrationality at the time rather than in hindsight.
One argument in the book that I didn't find entirely convincing is that riskier (with respect to nondiversifiable risk) stocks will have greater expected returns. This seems logical but Malkiel quotes a study which found otherwise and does not adequately explain why he doubts it.
In summary if you are unfamiliar with the case for index funds this book is probably a good introduction but more knowledgeable readers may find little new in it.