Why we shouldn’t combine Higgs searches across experiments.On July 4th, the CMS and ATLAS will present their current state of the Higgs searches. There is no reliable information on whether or not one of these experiments (or both) will claim the discovery of the Higgs boson – at least a 5-sigma signal.
The data collected by each experiment may be calculated to be approximately enough for a 5-sigma discovery right now so we will have to be surprised whether they have been "lucky" or not.
Some rumors coming from extremely unreliable places say that they have been lucky enough. Chances are not far from 50% that the rumors are right. However, if you believe that the 4.5-sigma combined excesses from 2011 show that the Higgs boson is there, it's guaranteed that there will be enough evidence to discover the Higgs boson in the combined charts that include the collisions recorded both inside the ATLAS detector as well as the CMS detector. I am confident that the combined significance level will be above 6 if not 7 sigma.
Aidan finds it inappropriate to combine the data. He argues as follows:
The next obvious step would be to combine the results from the two experiments and count the sigma. Despite being an obvious next step, this is the worst thing we could do at the moment. The Higgs field was postulated nearly 50 years ago, the LHC was proposed about 30 years ago, the experiments have been in design and development for about 20 years, and we’ve been taking data for about 18 months. Rushing to get a result a few weeks early is an act of impatience and frustration, and we should resist the temptation to get an answer now. Providing good quality physics results is more important than getting an answer we want.From these paragraphs, assuming that Aidan is a rank-and-file member who must obey his bosses, we are learning that ATLAS and CMS will not combine their charts on July 4th (although the two Tevatron experiments did exactly that when they officially discovered the top quark). However, I think that his explanations why the two experiments' data shouldn't be combined are pure emotions. The quality isn't lowered in any way if one combines the results.
It's actually the duty – the only right attitude – for any scientist outside CMS and ATLAS to combine the results. Why? Simply because a fair and impartial scientist must take all the existing evidence for a theory or against a theory into account. In total, the LHC has accumulated about 20 inverse femtobarns of proton-proton collisions at 7 or 8 TeV – that translates to 1.4 quadrillion collisions – in 2011 and 2012. These collisions may be evaluated by two different teams, for various reasons I will discuss momentarily. But all of them are valuable data and no impartial scientist outside these collaborations may justify any kind of cherry-picking.
Also, it's a red herring to say that we "want" the Higgs. I don't "want" the Higgs. I may be much happier if the LHC discovered the Motl fermion instead of the Higgs boson. I just see clear data showing that the Higgs boson is there and this certainty will probably become even more extreme on July 4th.
It's simply a fact that if ATLAS ended up with a 4.5 sigma signal and CMS would have the same, we could still combine them in quadrature to get a 6-sigma or 7-sigma excess. One may reconstruct the number of events in each channel – counting both the collisions inside ATLAS and inside CMS – and see that the evidence against the null hypothesis (a Standard Model without any nearby Higgs boson) has reached a 6-sigma confidence level (or higher).
It would be nothing else than a pure denial of scientific facts if someone would reject the claim that the higgsless hypothesis has been excluded with the certainty required by particle physics. One may offer lots of moralizing words but all of them are irrational gibberish. Including all the known evidence is the only honest and accurate enough way to give the best possible answer at this moment to the question whether the experiments show that there is a Higgs boson near 125 GeV.
If we combine measurements from two different experiments we end up losing the vital crosscheck. The best way to proceed is two wait a few more weeks until both experiments can produce a 5 sigma discovery and see if these results agree.This "crosscheck" comment is also irrational because if both experiments will show a near-5-sigma excess in similar channels (just imagine that none of them will get "there" separately), then the crosscheck will have been done and its result will have been "passed". There is no other "crosscheck" one needs to do when he determines that two experiments are saying pretty much the same thing about the existence of the Higgs boson. Waiting for another "crosscheck" and not knowing what this "crosscheck" is exactly supposed to be – except for seeing that the experiments agree within the error margins which will have been achieved – is nothing else than obstructionism, a nonsensical justification of the denial of one-half of the data.
Finally, I want to discuss the "experimental duality cult":
The reason we have two experiments at the LHC looking for the Higgs boson is because if one experiment makes a discovery then the other experiment can confirm or refute the discovery. This is why we have both D0 and CDF, both Belle and BaBar, both ATLAS and CMS, both UA2 and UA1 (where in the interest of fairness the orders the names are chosen at random.) Usually these pairs of experiments are neck and neck on any discovery or measurement, so when one experiment sees an effect but its counterpart doesn’t then it’s likely due to a problem with the analysis. Finding such a problem does not indicate poor scientific practice or incompetence, in fact it’s part of the scientific method to catch these little hiccups.This is just plain bullshit. There is nothing essential for the scientific method and its quality about the maintaining of such redundandies. The actual reason why there are two major detectors at the LHC, ATLAS and CMS, is that the price tag of the LHC ring was nearly $10 billion while one big detector only cost $1 billion or so. So it would be waste of space and an imbalanced investment if the $10 billion ring only harbored one detector.
Instead, people decided to build two detectors, to double the number of collisions. The remaining choice was whether the detectors would follow the same design (which would be a bit cheaper) or different designs. It just became fashionable sometime in the 1980s to have detectors of different designs on the same accelerator but there's nothing "vital" about this diversity, either. If the LHC hosted two ATLAS detectors or two CMS detectors instead of one ATLAS and one CMS, we would know pretty much the same thing today, with pretty much the same reliability and data quality.
A Time-Lapse Video Shows Creation of Giant Higgs Boson Mural: on the side of the ATLAS control room (2010)
It's a pure convention to build detectors according to two designs; and it's a pure convention to have two teams of physicists working on them. One could also have three or 47 detectors and 470 teams like that. But one could also live with one. In fact, most of the key discoveries in physics (and science) occurred without any such redundancy. There was no Rutherford II who would be discovering the nucleus; there was no Eddington II who would travel to a different forest to observe the 1919 solar eclipse; I could continue like that.
Before the 1980s, even particle physics worked without such redundancies. When SLAC was discovering quarks and QCD in the late 1960s (deep inelastic scattering), they weren't employing two competing groups of experimenters. And it just worked, too. One could argue that the redundancy of the experiments is just another manifestation of the reduced efficiency of the contemporary institutionalized science, an example showing how the money is wasted in the public sector, a demonstration that the Big Science also tends to behave in the same way as the Big Government and it tries to employ as many people as possible even though they're not necessary. The Big Government and bureaucracy – and the Big Science is unfortunately similar in many cases – is able to reduce the efficiency of the process to an arbitrarily low level.
All these people are employed – and someone responsible for that may think that he or she is "nice" because of that – but they inevitably end up somewhat frustrated because even when they discover the Higgs boson, and a similarly vital particle is only discovered a few times in a century, each of the ATLAS and CMS member will only "possess" 1/6,000 of such a discovery. She or he is an irrelevant wheel in an unnecessarily large machine. Everyone knows that. The members know that, too. It's a price for someone's decision to give "jobs" to a maximum number of physicists. It's not the only price, of course.
Someone wants to double the number of the people who are getting salaries from the public sector? No problem, why don't we just double the number of otherwise equivalent experiments?
Concerning the inefficiencies that automatically come together with the Big Government, one could also argue that there exists a deliberate effort to delay and slow down the discoveries – it's implicit even in Aidan's text – so that the experimenters don't look "useless" in the following year or years (assuming that nothing else will be discovered which is a huge assumption). But that's an unacceptable policy, too. If discoveries may be made quickly, it's just wrong to artificially slow them down. It's a bad testimony about the efficiency of the system if some people manifestly think that they will be rewarded if their work is extra slow.
Experiments may check each other but is this possibility to check the other folks' work really worth the money? I would say it's not. CDF claimed crazy things and bumps and D0 later rejected them. But if D0 hadn't done it, sensible people would have thought that CDF made an error, anyway (I have surely recognized all the bogus CDF papers as soon as they published them). And these claims would be pretty much experimentally rejected by the LHC by today, anyway.
Incidentally, I also strongly disagree with Aidan's claim that "Finding such a problem does not indicate poor scientific practice or incompetence, in fact it’s part of the scientific method to catch these little hiccups." The numerous erroneous claims that the CDF has made in recent years – "little hiccups" is an over-the-edge euphemism – surely do show that their research work was poor in recent years. If the redundancy and competition is good for anything, it's good exactly for knowing that D0 has done a better job in the 145 GeV bump and some bizarre top-quark-related claims than the CDF did. In other words, CDF did a poor job and if the errors boiled down to human errors, then the disproved observations by the CDF show someone's incompetence, too. The scientific method indeed expects people to search for errors in their and other people's work; but once such errors are found, it does mean something about the competence and all functional scientific institutions will inevitably reflect this competence or incompetence in hiring and funding decisions! This is a part of the separation of the wheat from the chaff that is so essential in science! If D0 is doing things in a better way than the CDF, it's desirable to strengthen "things and people working in the D0 way" and suppress "things and people working in the CDF way". Saying something else is a denial of the good management, denial of natural selection, denial of almost all mechanisms that improve the society in the long run.
One may also say that the redundancy is neither sufficient for a reliable crosscheck nor useful when the crosscheck fails. Even if there are two experiments, they may make an error which is especially likely if all the experimenters are affected by the same lore and habits (or the detectors are plagued by similar design flaws). And on the contrary, if one experiment debunks another, we really don't know who is right. So strictly speaking, when D0 rejected the CDF claims about the 145 GeV Wjj bump, we should have said that it was a conflict and we don't know who is right so having two detectors didn't really help us to learn the truth. To decide who is probably right, we need some "theoretical expectations", anyway. D0 is right because it's much more likely for someone (CDF) to make an error that deviates from the Standard Model in a random way – there are many such potential errors – than for someone else (D0) to make the right unique error that is needed to exactly cancel Nature's deviation from the Standard Model. But with similar "theoretical expectations", I didn't really need D0 to know that CDF was wrong, anyway.
The ongoing discussion about the "advantages of the competition" is much more general. Some people want to social-engineer competition in the markets. Whenever a company is successful and large or dominant in some business, they want to hurt it, cut it, destroy it, subsidize its competitors, and so on. The idea is that such an intervention improves the products and consumers will benefit etc. However, there is no valid general rule of this sort. In some cases, one company is simply doing a better job than others and that's why it's dominant. This dominant company knows the best recipe to transform an invested dollar to a profit in the future. So any different allocation than the allocation that makes it dominant is misallocation; by hurting the dominant company, it also reduces the profitability and efficiency of the system as a whole.
What makes capitalism superior isn't "forced competition" that is as diverse as possible; what makes capitalism superior is the "freedom for the competition to appear". But if the conditions are such that there is no competition and things can be produced more efficiently by one company which makes everyone satisfied, then this is how it should be (at least temporarily, but the market decides how much long this situation may last).
Imagine that you fund N otherwise (nearly) equivalent experiments and the question is for what positive value of N, you get the optimum ratio of "value of scientific information you get from the experiments" over "the price of the experiments". A simple question. Aidan clearly wants to suggest – with all the comments about "good quality" and "crosschecks" – that the best value is N=2. But that's rubbish, of course. The law of diminishing marginal returns implies that the optimum value of N is N=1. What the additional experiment gives you is inevitably less valuable than what the first one could have given you. In most cases, the experiments simply overlap so they must share the "scientific profit".
To summarize, the redundancy and diversity of the experiments is grossly overrated. By convention, people chose to evaluate the LHC data in two separate teams. It's a perfectly legitimate although not necessarily economically optimal design – like the design of a car with 6 or 8 cylinders. But this randomly chosen design shouldn't be worshiped as a key ingredience of the scientific method. It's certainly not one.
While ATLAS and CMS folks are supposed to work in isolation in order to check each other at the very end of the research, it's still true that the LHC has accumulated a certain amount of experimental data – 1.4 quadrillion collisions – and throwing away some of the data would mean to deny the experimental evidence. ATLAS and CMS employees may have internal rules not to look at other experimenters' data; that's just fine as an internal etiquette. But impartial scientists who are not constrained by similar internal rules just can't afford to ignore one-half of the collisions; or to pretend that the two sets of the collisions don't affect the answers to the same questions (i.e. pretend that the data can't be combined).
They definitely do. Whenever a physicist wants to know what the LHC is telling us today, he must combine all the collisions.
And that's the memo.
0 comments:
Post a Comment