Priors

Recently Dylan Farrow, the adopted daughter of Woody Allen and Mia Farrow, repeated allegations that Allen had sexually molested her when she was 7 years old.  These allegations were first made in 1982 during Allen and Farrow's acrimonious breakup (triggered by Farrow discovering Allen was involved with her adopted daughter Soon-Yi).  The allegations were investigated at the time and no criminal charges were pressed against Allen although the judge in a custody battle between Allen and Farrow seems to have given them some credence. 

The revived charges led to considerable internet discussion with some people quite convinced of Allen's guilt while others were far more skeptical.  I think it is worth noting that this disagreement can reflect different prior beliefs about how common sexual abuse of children is rather than different opinions about the strength of the evidence in Allen's case.

Suppose for example that some people believe that fathers rarely molest their children say .1% of the time while others believe it is fairly common say 10% of the time.  Suppose further that a credible accusation of abuse will be made in 1% of the cases where no abuse occurred and (for simplicity) in 100% of the cases where abuse did occur.  Then what fraction of credible accusations will in fact be true?

If sexual abuse is rare then out of every million men there will be 1000 cases of actual abuse (hence 1000 true accusations) and 9990 false accusations (.01 * 999000).  So only 9.1% of the credible accusations will actually be true.
On the other hand if abuse is common there will be 100000 true accusations and 9000 false accusations.  So in this case 91.7% of the accusations will be true.  Hence in a case like Allen's your prior beliefs can determine whether you see the evidence of guilt as weak or strong.

Of course I made up the above numbers to illustrate my point.  The reader can try different numbers and see how the fraction of false accusations changes.  I will note that assuming (more realistically) that less than 100% of actual abuse is reported leads to an increase in the fraction of false accusations.

This false positive problem also occurs in other contexts.  For example if a medical condition is rare a positive test result may be more likely test error than a true positive even if the test is fairly reliable.

My Citations

I recently discovered the "My Citations" feature on Google Scholar.  It is kind of neat.  You set up a profile with a list of your papers and then Google Scholar finds, counts and lists many of papers which have cited each of your papers.  Listing my papers was easy as Google had already grouped most of them together and I just had to add the whole group and then delete a few interlopers (you would think Google would realize a paper published in the 1700s didn't belong with the others even if it was also written by James Shearer).  In order to make your profile public (so it shows up when you search Google Scholar for your name) you need a verified email address at an academic institution.  This was a bit of a problem for me as I don't have a current academic affiliation.  But then I figured out as a MIT graduate I could set up an email address at alum.mit.edu which would just forward email to my personal account.  After I did this (and waited a day for it to take effect) I was able to use it to pass Google's verification and make my profile public.

When I was at IBM I tried for a while (until I lost interest) to keep track (via Science Citation Index) of papers citing mine.  This is much easier.  And Google Scholar seems to be pretty good at tracking down free online versions of published papers.  It is interesting to see that the number of people citing my papers each year has been trending upward although I haven't been producing much in the way of new papers for a long time.  However I suspect this is mostly because the number of papers being published each year has also been steadily rising.     

In summary this seems like an interesting and worthwhile feature.  It does depend on authors setting it up for their papers.  At the moment many haven't but hopefully more will over time.

Benoit Mandelbrot RIP

Benoit Mandelbrot died last week. I had lunch with him a few times when I worked at IBM and found him pleasant enough although I could see why some of my colleagues thought he was a bit full of himself.

Complexity theory update

I was asked in comments about the claimed P != NP proof I posted about last week. This NYT article has a reasonable summary of current opinion. Basically the "proof" appears to have some serious problems. Unfortunately the author may have trouble accepting this verdict.

As for what a proof would mean, the bare fact alone would not change much since many people already assume it is true. However it is likely a valid proof would depend on much greater insight into the theory computation than we currently have.

Alternating sequences

Last week I heard a talk by Richard Stanley in which he mentioned a version of the following problem. Define an alternating sequence as one in which the differences between adjacent elements are alternatively positive and negative. So 1,2,3 and 3,2,1 are not alternating sequences but 1,4,2,5,3,6 is. Consider random permutations of 1,2,...,n. We ask as n goes to infinity what is the expected length of the longest alternating subsequence? Note the elements of a subsequence do not have to be adjacent in the original sequence. You can think of a subsequence as what is left after you delete some elements of a sequence.

Big news in complexity theory?

An HP researcher, Vinay_Deolalikar , is claiming to have proved that P != NP. This is of course a major result if correct. I would guess the odds are it isn't but an expert thinks it is worth taking seriously. I glanced through the preliminary version of the paper. It goes well beyond my superficial knowledge of complexity theory so I will defer to more expert opinion as to correctness. I expect this will be forthcoming fairly quickly.

NYT AIDS story

I think the New York Times is a pretty good newspaper in general but this story on an AIDS prevention gel for women is just dreadful. It starts:

The best AIDS-prevention news in years was released here last week at a world conference on the disease: a vaginal gel, called a microbicide, that can be used without a man knowing it, gave women a 39 percent chance of avoiding infection with the deadly virus.

Thirty-nine percent is, obviously, not perfect, though the women in the South African trial who used the gel most faithfully did better, achieving 54 percent protection.

It is completely unclear what the 39% and 54% are referring to and this is not explained in the remainder of the story. I suspect this is because the reporter had no clue. But if he can't explain what these numbers mean they shouldn't be in the story.

Rounding error

Here is an easy math puzzle. Suppose we have 3 random variables a,b and c with c=a+b. We round a to A, b to B and c to C and want to know the chances that C=A+B. We need some more information for an unique answer. So for definiteness let a and b be random real variables uniformly distributed between 0 and 50. Let c=a+b and let A,B and C be a,b and c rounded to the nearest integer (it won't matter how .5 cases are rounded). Then we ask what is the probability that C=A+B?

This problem was inspired by the recent furor regarding the Research 2000 polling operation in which it has been suggested that Research 2000 was saving money by making up its poll results. However it is unclear if the above problem actually has any connection to the anomalies that have been found in the Research 2000 results.

Paint problem

Here is a nice math problem I encountered recently.

N painters arrive at random positions around a circular fence. Each paints the section of fence between them self and the nearest neighboring painter. As a result of this curious procedure some sections of fence are painted twice and some are not painted at all. In the limit as N goes to infinity what fraction of the fence on average will be painted (at least once)?

Actually as originally posed the fence was not circular which doesn't change the limit but is not as easy to deal with.

You can also ask the same question for the case where each painter paints the section of fence between them self and the farther of the two adjacent painters.

Martin Gardner RIP

Martin Gardner died last Saturday . He wrote the Mathematical Games column in Scientific American from 1956 to 1981 and several books of collected columns (as well as many other books). I enjoyed his recreational mathematics works while growing up and they probably influenced me towards a career in mathematics.

Stopping distance puzzle.

Here is a quick puzzle. It is pretty easy so you might try computing the answer in your head. Suppose you are driving at 60 mph and apply the brakes. Suppose you can brake with a force of 1g. How far will you travel before stopping? For this puzzle assume 1g = 32 ft/(sec)/(sec).

Added note: As discussed in the comments I was a bit sloppy when I said "brake with a force of 1g". What I meant of course was brake so as to produce a deceleration of 1g. See this wikipedia article for much more about this common misnomer.

Lottery tickets

Another comment on "The Black Swan". In the glossary on pages 308-309 Taleb defines:

Lottery-ticket fallacy: the naive analogy equating an investment in collecting positive Black Swans to the accumulation of lottery tickets. Lottery tickets are not scalable.

By scalable Taleb means having unlimited upside.

The analogy to lottery tickets is unwelcome to Taleb because he advocates a strategy of betting on long shots which you are hoping will pay off because of unexpected events ("Black Swans" in his terminology). This looks something like buying lottery tickets but buying lottery tickets is generally a stupid strategy. The reason is not because they don't have unlimited upside (the millionfold gains available in typical state lotteries are greater than are plausibly possible in most other venues) but because the tickets cost too much. The New York State lottery seems to return about 40% of the amount bet which I think is fairly typical. So if you could buy lottery tickets for 10 cents on the dollar they would be a fine investment.

Taleb believes markets don't properly price in the possibility of extreme events, Black Swans. This is plausible but there is a problem in trying to take advantage of such mispricing. Namely, as the example of lotteries shows, people like making long shot bets. So any particular long shot bet may be overpriced (as lottery tickets are). So you can't just blindly bet on long shots (lest you violate another of Taleb's precepts which is don't be the sucker) you have to locate good long shot bets. But Taleb doesn't offer much help in that regard.

Forecast spread

I have been reading "The Black Swan" by Nassim Nicholas Taleb. It is written in a kind of arrogant style. I don't find it all that annoying perhaps because I largely agree with the book (at least so far). However it does encourage my natural inclination to nitpick.

On page 150-151 Taleb claims:

... Worse yet, the forecaster's errors were significantly larger than the average difference between individual forecasts, which indicates herding. Normally, forecasts should be as far from one another as they are from the predicted number. ...

This claim which I believe I have seen elsewhere is just not true. There is no reason to expect even independent unbiased predictions to behave in this way. Suppose for example we ask for forecasters to predict how many heads will be observed in 100 flips of a fair coin. Naturally they will all predict 50 but that doesn't mean we can expect exactly 50 heads. This is an artificial example but you will see the same clustering whenever there is a large random component in the variable being forecast. This variable could be for example the number of homicides in New York in 2010 or the size of the corn harvest in Iowa. Forecasters will be in effect estimating the mean of an underlying random distribution and there is no reason to expect these estimates to be as widely spread as the distribution itself.

Going for two

Sunday I was listening to the Giants-Falcons game on the radio when the Falcons, trailing by 14, scored a touchdown with about six minutes left in the game. Playing for overtime the Falcons kicked the extra point. They scored another touchdown but then lost in overtime. There was a case for playing to win in regulation by going for the two point conversion after the first touchdown. Suppose we assume the Falcons will score another touchdown and the Giants will not score again in regulation. Suppose the chance of converting a two point conversion try is p and the chance of making an extra point try is q and that if the game goes into overtime each team has a 50% chance to win. Then what is the relationship between p and q that determines whether it is better for the Falcons to go for the two point conversion or try to kick the extra point after their first touchdown?

Tax efficiency

I have just spent a couple of days entering data for some mutual funds I own into my new Quicken program . I had been vaguely aware that index funds have tax advantages over actively managed mutual funds but a concrete example was still startling. I owned $X of index fund A and $Y of actively managed fund B on 12/31/1986. In both cases I have reinvested all distributions and paid the taxes with other income. As of 10/28/2009, A was worth 7.21*X and B was worth 6.69*Y. So ignoring taxes the annual rates of return for A and B are about 9.04% and 8.69% respectively. The higher fees of active management don't seem to be adding value. However the after tax picture is even worse. Over the years the index fund has distributed 2.28*X while the actively managed fund has distributed 8.01*Y. So the tax cost of B has been much higher than A. This probably has increased the after tax difference in annual return by 1% or so.

Now much of this tax advantage would disappear if I sold both funds as the distributions have increased the basis so selling A would produce a substantial capital gain and selling B would produce a substantial capital loss. However under current law if I hold until death the basis will step up preserving the tax advantage.

Another way of looking at this is to assume annual returns of 9% consisting of 3% in dividends and 6% in capital appreciation. Suppose the index fund just distributes the dividends while the actively managed fund distributes the dividends and the capital gains. Then if you assume a tax rate of 20% and that you pay the taxes from the distributions and reinvest the remainder, the index fund will grow at an annual rate of 8.4% while the annual rate of growth for the actively managed fund will be 7.2%.

To substantially benefit from tax free compounding rather long holding periods are required. Suppose we ignore dividends and assume capital appreciation of 6% a year. Assume we hold for n years, then sell and pay 20% capital gains tax and reinvest. Then as n goes to infinity the effective annual yield rises from 4.8% (n=1) to 6% but rather slowly. It is an interesting exercise to determine how big n is required to be to get half the benefit (ie an annual yield of 5.4%).

Playoff puzzle

There was some angst in Yankeeland after the team failed to close out the Angels in game 5. However the Yankees would appear to still be in pretty good shape returning home up 3 games to 2 in the best of 7 American League Championship Series (ALCS).

This suggests the following puzzle. Suppose the Yankees will win each game with probability p (independent of the results of the other games). Then in a best of 7 playoff how large does p have to be for the Yankees to be worse off leading 3 to 2 than they were at the beginning before any games were played.

Mathematics and CDOs

This mathematics paper (with this FAQ ) has attracted some attention from bloggers. Briefly the paper starts with a model in which sellers construct collateralized debt obligations (or CDOs) from many different asset types (and with each asset type appearing in many different CDOs). If the seller knows some of the asset types are lemons (eg are worse risks than they appear) then the seller can create lemon CDOs by assigning them more than their fair share of the lemon asset types. The paper claims to show (using the conjectured intractability of the densest subgraph problem) that this nonrandom assignment is much easier for the seller to do than for the buyer to detect.

I have no reason to doubt the mathematics in this paper but I do doubt its real world significance. The larger risk with such CDOs from the buyer's point of view is that the CDOs will be overpriced across the board not that they will purchase CDOs from a malevolently constructed fraction of lemons. This is what went wrong with the mortgage backed CDOs whose loss of value contributed so much to the recent crisis. The pricing models for these CDOs overvalued them by incorporating a number of overly optimistic assumptions. When this became apparent their prices collapsed. As far as I know the scenario in paper is purely hypothetical.

I don't see much point in worrying about hypothetical problems before fixing readily apparent real problems. And I don't think this particular hypothetical problem would actually be all that difficult to solve. It seems to me that it should be possible to devise a transparent verifiable randomization procedure if that should be felt to be important.

My opinion is this problem will prove to be largely moot going forward. These CDO products grew explosively because they could be sold for more than they were worth. Presumably potential buyers have wised up and this will no longer be possible. In which case I don't think these products have much reason for existing and will largely disappear.

Mathematics

I was asked in comments a couple of days ago whether I was doing mathematics while out of work. Somewhat sad to say, the answer is no or at least not seriously. While with my former employer I gradually did less research level mathematics until in the final years I was doing programming almost exclusively. So at this point I am rather out of touch with the current research and am not very motivated to get back into it.

Some people can work for years on a problem without much interaction with other people. I am not really able to do this. So as I gradually lost contact with the mathematics research community I became less productive. And I became more interested in other things.

Vaccine trial statistics

The US military has been conducting a large and expensive AIDS vaccine trial in Thailand. Recently positive results were announced with much fanfare .

However the actual results are not that impressive. 74 out of about 8000 in the placebo group became infected as opposed to 51 out of about 8000 in the vaccine group. An approximate test of significance is to ask if we flip a fair coin 125 times what percentage of the time will the head tail split be at least as uneven as 74/51. By my calculation 4.87%. So this is statistically significant at the 5% level but just barely so. Again by my calculations if there had been one fewer infection in the placebo group this value would have been 5.89% or if there had been one more infection in the vaccine group this value would have been 6.09% neither of which would have been significant.

This illustrates one of the problems of picking an arbitrary line like 5% and declaring results on one side successes and on the other side failures. Another problem is the fact that there have been many vaccine trials other than this one all of which failed. Clearly if you run enough trials some will appear to succeed (at the 5% level) just by chance. I think there is a fair chance this is an example.

Annuity math

As I noted earlier I plan to start receiving a pension from my former employer in November after I turn 55. For obscure reasons I have options to take portions as a lump sum instead of a single life annuity. In one case the lump sum is about 90 times the monthly annuity and in the other case it is about 133. The information packet I received suggested these options are bad deals and that a fair conversion ratio would be about 188.5. This is roughly consistent with annuity quotes I found on the web which had ratios which varied from 164 to 206.

According to the information packet a life expectancy (for males) at 55 of 26 years and a discount rate of 4.3% was used to compare the value of the annuity and lump sum options. And indeed assuming I will live exactly 26 additional years (receiving 312 monthly payments) and applying a discount rate of 4.3% does produce a similar ratio, 189.3. The discount rate needed to produce a ratio of 133 seems to be over 8%. Since I don't think my expected investment returns are over 8% and I don't have any reason to believe my life expectancy is significantly less than average it looks like I will decline the lump sum options.