How do we know the LHC results are robust?Where can we find LHC results talks online?If the LHC-calculated...
Should my PhD thesis be submitted under my legal name?
Products and sum of cubes in Fibonacci
Opposite of a diet
Is it okay / does it make sense for another player to join a running game of Munchkin?
Bash method for viewing beginning and end of file
Confused about a passage in Harry Potter y la piedra filosofal
Will it be accepted, if there is no ''Main Character" stereotype?
Valid Badminton Score?
Is HostGator storing my password in plaintext?
Hide Select Output from T-SQL
Can I use my Chinese passport to enter China after I acquired another citizenship?
Using parameter substitution on a Bash array
The baby cries all morning
What't the meaning of this extra silence?
Is there an Impartial Brexit Deal comparison site?
Is the destination of a commercial flight important for the pilot?
Is this Spell Mimic feat balanced?
Curses work by shouting - How to avoid collateral damage?
How was Earth single-handedly capable of creating 3 of the 4 gods of chaos?
Your magic is very sketchy
What defines a dissertation?
What is the intuitive meaning of having a linear relationship between the logs of two variables?
Time travel short story where a man arrives in the late 19th century in a time machine and then sends the machine back into the past
Failed to fetch jessie backports repository
How do we know the LHC results are robust?
Where can we find LHC results talks online?If the LHC-calculated mass of the Higgs is wrong, how long will it take to determine this with confidence?How are the two proton beams at the LHC accelerated in opposite directions?How are the protons for collisions in the LHC made?How the LHC bump can be a mere coincidence?How does the LHC explore extra dimensions?Results of photon-photon collisions at LHC?How are anti quarks produced in $pp$ collision at LHC?What are the consequences of LHC results for supersymmetry?How does the LHC separate the protium isotope to have only protons for the collision?
$begingroup$
Nature article on reproducibility in science.
According to that article, a (surprisingly) large number of experiments aren't reproducible, or at least there have been failed attempted reproductions. In one of the figures, it's said that 70% of scientists in physics & engineering have failed to reproduce someone else's results, and 50% have failed to reproduce their own.
Clearly, if something cannot be reproduced, its veracity is called into question. Also clearly, because there's only one particle accelerator with the power of the LHC in the world, we aren't able to independently reproduce LHC results; in fact naively one might expect there's a 50% chance that if we built another LHC, it will not reach the same results. How, then, do we know that the LHC results (such as the discovery of the Higgs boson) are robust? Or do we not know the LHC results are robust, and are effectively proceeding on faith that they are?
particle-physics large-hadron-collider
$endgroup$
add a comment |
$begingroup$
Nature article on reproducibility in science.
According to that article, a (surprisingly) large number of experiments aren't reproducible, or at least there have been failed attempted reproductions. In one of the figures, it's said that 70% of scientists in physics & engineering have failed to reproduce someone else's results, and 50% have failed to reproduce their own.
Clearly, if something cannot be reproduced, its veracity is called into question. Also clearly, because there's only one particle accelerator with the power of the LHC in the world, we aren't able to independently reproduce LHC results; in fact naively one might expect there's a 50% chance that if we built another LHC, it will not reach the same results. How, then, do we know that the LHC results (such as the discovery of the Higgs boson) are robust? Or do we not know the LHC results are robust, and are effectively proceeding on faith that they are?
particle-physics large-hadron-collider
$endgroup$
$begingroup$
I think it is worth pointing out that the LHC doesn't just do one particle collision and then say the experiment is completed. How much do you know about what goes into such experiments, how many times they are actually repeated, and then how the data is analyzed from there?
$endgroup$
– Aaron Stevens
1 hour ago
$begingroup$
@AaronStevens I know some of it, but am not an expert. I know the LHC crashes two protons into each other multiple times, and the results of each collision are expected to be different but have different probabilities. Many of the daughter particles are unstable and also expected to decay. The detector sees the "final" products when they reach the detector, and the analysis is supposed to infer based on these detected particles what the original particles are. Does that answer your question?
$endgroup$
– Allure
56 mins ago
$begingroup$
I was asking if you had looked into the efforts taken to make sure the results from the LHC are good results and not just mistakes. Also the LHC isn't the only particle collider in existence.
$endgroup$
– Aaron Stevens
53 mins ago
$begingroup$
Related video
$endgroup$
– Aaron Stevens
42 mins ago
$begingroup$
For more about this important question than you may have bargained for, look for discussions about the "look-elsewhere effect" in the statistical analysis of the data from the mostly-independent ATLAS and CMS experiments at the LHC, especially in the context of their joint discovery of the Higgs particle.
$endgroup$
– rob♦
19 mins ago
add a comment |
$begingroup$
Nature article on reproducibility in science.
According to that article, a (surprisingly) large number of experiments aren't reproducible, or at least there have been failed attempted reproductions. In one of the figures, it's said that 70% of scientists in physics & engineering have failed to reproduce someone else's results, and 50% have failed to reproduce their own.
Clearly, if something cannot be reproduced, its veracity is called into question. Also clearly, because there's only one particle accelerator with the power of the LHC in the world, we aren't able to independently reproduce LHC results; in fact naively one might expect there's a 50% chance that if we built another LHC, it will not reach the same results. How, then, do we know that the LHC results (such as the discovery of the Higgs boson) are robust? Or do we not know the LHC results are robust, and are effectively proceeding on faith that they are?
particle-physics large-hadron-collider
$endgroup$
Nature article on reproducibility in science.
According to that article, a (surprisingly) large number of experiments aren't reproducible, or at least there have been failed attempted reproductions. In one of the figures, it's said that 70% of scientists in physics & engineering have failed to reproduce someone else's results, and 50% have failed to reproduce their own.
Clearly, if something cannot be reproduced, its veracity is called into question. Also clearly, because there's only one particle accelerator with the power of the LHC in the world, we aren't able to independently reproduce LHC results; in fact naively one might expect there's a 50% chance that if we built another LHC, it will not reach the same results. How, then, do we know that the LHC results (such as the discovery of the Higgs boson) are robust? Or do we not know the LHC results are robust, and are effectively proceeding on faith that they are?
particle-physics large-hadron-collider
particle-physics large-hadron-collider
asked 1 hour ago
AllureAllure
1,991722
1,991722
$begingroup$
I think it is worth pointing out that the LHC doesn't just do one particle collision and then say the experiment is completed. How much do you know about what goes into such experiments, how many times they are actually repeated, and then how the data is analyzed from there?
$endgroup$
– Aaron Stevens
1 hour ago
$begingroup$
@AaronStevens I know some of it, but am not an expert. I know the LHC crashes two protons into each other multiple times, and the results of each collision are expected to be different but have different probabilities. Many of the daughter particles are unstable and also expected to decay. The detector sees the "final" products when they reach the detector, and the analysis is supposed to infer based on these detected particles what the original particles are. Does that answer your question?
$endgroup$
– Allure
56 mins ago
$begingroup$
I was asking if you had looked into the efforts taken to make sure the results from the LHC are good results and not just mistakes. Also the LHC isn't the only particle collider in existence.
$endgroup$
– Aaron Stevens
53 mins ago
$begingroup$
Related video
$endgroup$
– Aaron Stevens
42 mins ago
$begingroup$
For more about this important question than you may have bargained for, look for discussions about the "look-elsewhere effect" in the statistical analysis of the data from the mostly-independent ATLAS and CMS experiments at the LHC, especially in the context of their joint discovery of the Higgs particle.
$endgroup$
– rob♦
19 mins ago
add a comment |
$begingroup$
I think it is worth pointing out that the LHC doesn't just do one particle collision and then say the experiment is completed. How much do you know about what goes into such experiments, how many times they are actually repeated, and then how the data is analyzed from there?
$endgroup$
– Aaron Stevens
1 hour ago
$begingroup$
@AaronStevens I know some of it, but am not an expert. I know the LHC crashes two protons into each other multiple times, and the results of each collision are expected to be different but have different probabilities. Many of the daughter particles are unstable and also expected to decay. The detector sees the "final" products when they reach the detector, and the analysis is supposed to infer based on these detected particles what the original particles are. Does that answer your question?
$endgroup$
– Allure
56 mins ago
$begingroup$
I was asking if you had looked into the efforts taken to make sure the results from the LHC are good results and not just mistakes. Also the LHC isn't the only particle collider in existence.
$endgroup$
– Aaron Stevens
53 mins ago
$begingroup$
Related video
$endgroup$
– Aaron Stevens
42 mins ago
$begingroup$
For more about this important question than you may have bargained for, look for discussions about the "look-elsewhere effect" in the statistical analysis of the data from the mostly-independent ATLAS and CMS experiments at the LHC, especially in the context of their joint discovery of the Higgs particle.
$endgroup$
– rob♦
19 mins ago
$begingroup$
I think it is worth pointing out that the LHC doesn't just do one particle collision and then say the experiment is completed. How much do you know about what goes into such experiments, how many times they are actually repeated, and then how the data is analyzed from there?
$endgroup$
– Aaron Stevens
1 hour ago
$begingroup$
I think it is worth pointing out that the LHC doesn't just do one particle collision and then say the experiment is completed. How much do you know about what goes into such experiments, how many times they are actually repeated, and then how the data is analyzed from there?
$endgroup$
– Aaron Stevens
1 hour ago
$begingroup$
@AaronStevens I know some of it, but am not an expert. I know the LHC crashes two protons into each other multiple times, and the results of each collision are expected to be different but have different probabilities. Many of the daughter particles are unstable and also expected to decay. The detector sees the "final" products when they reach the detector, and the analysis is supposed to infer based on these detected particles what the original particles are. Does that answer your question?
$endgroup$
– Allure
56 mins ago
$begingroup$
@AaronStevens I know some of it, but am not an expert. I know the LHC crashes two protons into each other multiple times, and the results of each collision are expected to be different but have different probabilities. Many of the daughter particles are unstable and also expected to decay. The detector sees the "final" products when they reach the detector, and the analysis is supposed to infer based on these detected particles what the original particles are. Does that answer your question?
$endgroup$
– Allure
56 mins ago
$begingroup$
I was asking if you had looked into the efforts taken to make sure the results from the LHC are good results and not just mistakes. Also the LHC isn't the only particle collider in existence.
$endgroup$
– Aaron Stevens
53 mins ago
$begingroup$
I was asking if you had looked into the efforts taken to make sure the results from the LHC are good results and not just mistakes. Also the LHC isn't the only particle collider in existence.
$endgroup$
– Aaron Stevens
53 mins ago
$begingroup$
Related video
$endgroup$
– Aaron Stevens
42 mins ago
$begingroup$
Related video
$endgroup$
– Aaron Stevens
42 mins ago
$begingroup$
For more about this important question than you may have bargained for, look for discussions about the "look-elsewhere effect" in the statistical analysis of the data from the mostly-independent ATLAS and CMS experiments at the LHC, especially in the context of their joint discovery of the Higgs particle.
$endgroup$
– rob♦
19 mins ago
$begingroup$
For more about this important question than you may have bargained for, look for discussions about the "look-elsewhere effect" in the statistical analysis of the data from the mostly-independent ATLAS and CMS experiments at the LHC, especially in the context of their joint discovery of the Higgs particle.
$endgroup$
– rob♦
19 mins ago
add a comment |
2 Answers
2
active
oldest
votes
$begingroup$
That's a really great question. The 'replication crisis' is that many effects in social sciences couldn't be reproduced. There are many factors leading to this phenomenon, including
- Weak standards of evidence, e.g., $2sigma$ evidence required to demonstrate an effect
- Researchers (subconsciously or otherwise) conducting bad scientific practice by selectively reporting and publishing significant results. E.g. considering many different effects until they find a significant effect or collecting data until they find a significant effect.
- Poor training in statistical methods.
I'm not entirely sure about the exact efforts that the LHC experiments are making to ensure that they don't suffer the same problems. But let me say some things that should at least put your mind at ease:
- Particle physics typically requires a high-standard of evidence for discoveries ($5sigma$)
- The results at the LHC are already replicated!
- There are several detectors placed around the LHC ring. Two them, called ATLAS and CMS, are general purpose detectors for Standard Model and Beyond the Standard Model physics. Both of them found compelling evidence for the Higgs boson.
- The Tevatron, a similar collider experiment in the USA operating at lower-energies, found direct evidence for the Higgs boson.
- The Higgs boson was observed several datasets collected at the LHC
- The LHC (typically) publishes findings regardless of their statistical significance, i.e., significant results are not selectively reported.
- The LHC teams are guided by statistical committees, hopefully ensuring good practice
- The LHC is in principle committed to open data, which means a lot of the data should at some point become public. This is one recommendation for helping the crisis in social sciences.
- Typical training for experimentalists at the LHC includes basic statistics (although in my experience LHC experimentalits are still subject to the same traps and misinterpretations as everyone else).
- All members (thousands) of the experimental teams are authors on the papers. The incentive for bad practices such as $p$-hacking is presumably slightly lowered, as you cannot 'discover' a new effect and publish it only under your own name, and have improved job/grant prospects. This incentive might be a factor in the replication crisis in social sciences.
- All papers are subject to internal review (which I understand to be quite rigorous) as well as external review by a journal
- LHC analyses are often (I'm not sure who plans or decides this) blinded. This means that the experimentalists cannot tweak the analyses depending on the result. They are 'blind' to the result, make their choices, then unblind it only at the end. This should help prevent $p$-hacking
If anything, there is a suspicion that the practices at the LHC might even result in the opposite of the 'replication crisis;' analyses that find effects that are somewhat significant might be examined and tweaked until they decrease.
$endgroup$
add a comment |
$begingroup$
This is an answer to the question above, as I do not have access to the paper I cannot really discuss it.
Here is a simpler example: Take an Olympics athlete. How many failures before breaking the record? Is the record not broken because there may have been a thousand failures before breaking it?
What about the hundreds of athletes who try to reproduce and get a better record? Should they not try?
The statistics of failed experiments is similar: There is a goal (actually thousands of goals depending on the physics discipline), and a number of trials to reach the goal.
We know the LHC results are robust because there are two major and many smaller experiments trying for the same goals. The reason there are two experiments is so that systematic errors in one will not give spurious results. We trust that the measurement statistics that give the end results are correct, as we trust for the record breaking run that the measured times and distances are correct.
(And LHC is not an experiment. It is where experiments can be carried out depending on the efforts and ingenuity of researchers, it is the field where the Olympics takes place.)
The robustness of scientific results depends on the specific experimental measurements, not on integrating over all disparate experiments ever made. Bad use of statistics in mixing up apples and oranges and cauliflower and then applying it to the stock market.
Another way to look at it: If there were no failures , would the experiments mean anything? They would be predictable by pen and paper.
$endgroup$
2
$begingroup$
I'm not sure I buy the Olympics analogy. Failed attempts at breaking a record isn't the same thing as a failed attempt to reproduce an experiment. It also sounds like you are saying we should just cherry pick what does work and ignore when it fails.
$endgroup$
– Aaron Stevens
57 mins ago
$begingroup$
@AaronStevens It is the same as failed atempts for another runner to reproduce the record. I will edit, thanks
$endgroup$
– anna v
46 mins ago
$begingroup$
@AaronStevens " cherry pick what does work" but is not that evolution in general? and "ignore when it fails" one learns from failure to design better experiments.
$endgroup$
– anna v
37 mins ago
$begingroup$
and should you always expect no errors both in instruments and logic from experimenters? Science started with trial and error and evolved..Failures fall on the side, if not reproducible.
$endgroup$
– anna v
35 mins ago
1
$begingroup$
You're just making it sound like if just try hard enough we can reproduce anything (break the record) , but this is not always the case. Not being able to reproduce experiment A could mean that something was wrong with experiment A and that we shouldn't try to make too many predictions (or any at all) based on experiment A. We wouldn't want to keep chasing it (trying to break the record)
$endgroup$
– Aaron Stevens
23 mins ago
|
show 5 more comments
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "151"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
noCode: true, onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fphysics.stackexchange.com%2fquestions%2f468891%2fhow-do-we-know-the-lhc-results-are-robust%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
That's a really great question. The 'replication crisis' is that many effects in social sciences couldn't be reproduced. There are many factors leading to this phenomenon, including
- Weak standards of evidence, e.g., $2sigma$ evidence required to demonstrate an effect
- Researchers (subconsciously or otherwise) conducting bad scientific practice by selectively reporting and publishing significant results. E.g. considering many different effects until they find a significant effect or collecting data until they find a significant effect.
- Poor training in statistical methods.
I'm not entirely sure about the exact efforts that the LHC experiments are making to ensure that they don't suffer the same problems. But let me say some things that should at least put your mind at ease:
- Particle physics typically requires a high-standard of evidence for discoveries ($5sigma$)
- The results at the LHC are already replicated!
- There are several detectors placed around the LHC ring. Two them, called ATLAS and CMS, are general purpose detectors for Standard Model and Beyond the Standard Model physics. Both of them found compelling evidence for the Higgs boson.
- The Tevatron, a similar collider experiment in the USA operating at lower-energies, found direct evidence for the Higgs boson.
- The Higgs boson was observed several datasets collected at the LHC
- The LHC (typically) publishes findings regardless of their statistical significance, i.e., significant results are not selectively reported.
- The LHC teams are guided by statistical committees, hopefully ensuring good practice
- The LHC is in principle committed to open data, which means a lot of the data should at some point become public. This is one recommendation for helping the crisis in social sciences.
- Typical training for experimentalists at the LHC includes basic statistics (although in my experience LHC experimentalits are still subject to the same traps and misinterpretations as everyone else).
- All members (thousands) of the experimental teams are authors on the papers. The incentive for bad practices such as $p$-hacking is presumably slightly lowered, as you cannot 'discover' a new effect and publish it only under your own name, and have improved job/grant prospects. This incentive might be a factor in the replication crisis in social sciences.
- All papers are subject to internal review (which I understand to be quite rigorous) as well as external review by a journal
- LHC analyses are often (I'm not sure who plans or decides this) blinded. This means that the experimentalists cannot tweak the analyses depending on the result. They are 'blind' to the result, make their choices, then unblind it only at the end. This should help prevent $p$-hacking
If anything, there is a suspicion that the practices at the LHC might even result in the opposite of the 'replication crisis;' analyses that find effects that are somewhat significant might be examined and tweaked until they decrease.
$endgroup$
add a comment |
$begingroup$
That's a really great question. The 'replication crisis' is that many effects in social sciences couldn't be reproduced. There are many factors leading to this phenomenon, including
- Weak standards of evidence, e.g., $2sigma$ evidence required to demonstrate an effect
- Researchers (subconsciously or otherwise) conducting bad scientific practice by selectively reporting and publishing significant results. E.g. considering many different effects until they find a significant effect or collecting data until they find a significant effect.
- Poor training in statistical methods.
I'm not entirely sure about the exact efforts that the LHC experiments are making to ensure that they don't suffer the same problems. But let me say some things that should at least put your mind at ease:
- Particle physics typically requires a high-standard of evidence for discoveries ($5sigma$)
- The results at the LHC are already replicated!
- There are several detectors placed around the LHC ring. Two them, called ATLAS and CMS, are general purpose detectors for Standard Model and Beyond the Standard Model physics. Both of them found compelling evidence for the Higgs boson.
- The Tevatron, a similar collider experiment in the USA operating at lower-energies, found direct evidence for the Higgs boson.
- The Higgs boson was observed several datasets collected at the LHC
- The LHC (typically) publishes findings regardless of their statistical significance, i.e., significant results are not selectively reported.
- The LHC teams are guided by statistical committees, hopefully ensuring good practice
- The LHC is in principle committed to open data, which means a lot of the data should at some point become public. This is one recommendation for helping the crisis in social sciences.
- Typical training for experimentalists at the LHC includes basic statistics (although in my experience LHC experimentalits are still subject to the same traps and misinterpretations as everyone else).
- All members (thousands) of the experimental teams are authors on the papers. The incentive for bad practices such as $p$-hacking is presumably slightly lowered, as you cannot 'discover' a new effect and publish it only under your own name, and have improved job/grant prospects. This incentive might be a factor in the replication crisis in social sciences.
- All papers are subject to internal review (which I understand to be quite rigorous) as well as external review by a journal
- LHC analyses are often (I'm not sure who plans or decides this) blinded. This means that the experimentalists cannot tweak the analyses depending on the result. They are 'blind' to the result, make their choices, then unblind it only at the end. This should help prevent $p$-hacking
If anything, there is a suspicion that the practices at the LHC might even result in the opposite of the 'replication crisis;' analyses that find effects that are somewhat significant might be examined and tweaked until they decrease.
$endgroup$
add a comment |
$begingroup$
That's a really great question. The 'replication crisis' is that many effects in social sciences couldn't be reproduced. There are many factors leading to this phenomenon, including
- Weak standards of evidence, e.g., $2sigma$ evidence required to demonstrate an effect
- Researchers (subconsciously or otherwise) conducting bad scientific practice by selectively reporting and publishing significant results. E.g. considering many different effects until they find a significant effect or collecting data until they find a significant effect.
- Poor training in statistical methods.
I'm not entirely sure about the exact efforts that the LHC experiments are making to ensure that they don't suffer the same problems. But let me say some things that should at least put your mind at ease:
- Particle physics typically requires a high-standard of evidence for discoveries ($5sigma$)
- The results at the LHC are already replicated!
- There are several detectors placed around the LHC ring. Two them, called ATLAS and CMS, are general purpose detectors for Standard Model and Beyond the Standard Model physics. Both of them found compelling evidence for the Higgs boson.
- The Tevatron, a similar collider experiment in the USA operating at lower-energies, found direct evidence for the Higgs boson.
- The Higgs boson was observed several datasets collected at the LHC
- The LHC (typically) publishes findings regardless of their statistical significance, i.e., significant results are not selectively reported.
- The LHC teams are guided by statistical committees, hopefully ensuring good practice
- The LHC is in principle committed to open data, which means a lot of the data should at some point become public. This is one recommendation for helping the crisis in social sciences.
- Typical training for experimentalists at the LHC includes basic statistics (although in my experience LHC experimentalits are still subject to the same traps and misinterpretations as everyone else).
- All members (thousands) of the experimental teams are authors on the papers. The incentive for bad practices such as $p$-hacking is presumably slightly lowered, as you cannot 'discover' a new effect and publish it only under your own name, and have improved job/grant prospects. This incentive might be a factor in the replication crisis in social sciences.
- All papers are subject to internal review (which I understand to be quite rigorous) as well as external review by a journal
- LHC analyses are often (I'm not sure who plans or decides this) blinded. This means that the experimentalists cannot tweak the analyses depending on the result. They are 'blind' to the result, make their choices, then unblind it only at the end. This should help prevent $p$-hacking
If anything, there is a suspicion that the practices at the LHC might even result in the opposite of the 'replication crisis;' analyses that find effects that are somewhat significant might be examined and tweaked until they decrease.
$endgroup$
That's a really great question. The 'replication crisis' is that many effects in social sciences couldn't be reproduced. There are many factors leading to this phenomenon, including
- Weak standards of evidence, e.g., $2sigma$ evidence required to demonstrate an effect
- Researchers (subconsciously or otherwise) conducting bad scientific practice by selectively reporting and publishing significant results. E.g. considering many different effects until they find a significant effect or collecting data until they find a significant effect.
- Poor training in statistical methods.
I'm not entirely sure about the exact efforts that the LHC experiments are making to ensure that they don't suffer the same problems. But let me say some things that should at least put your mind at ease:
- Particle physics typically requires a high-standard of evidence for discoveries ($5sigma$)
- The results at the LHC are already replicated!
- There are several detectors placed around the LHC ring. Two them, called ATLAS and CMS, are general purpose detectors for Standard Model and Beyond the Standard Model physics. Both of them found compelling evidence for the Higgs boson.
- The Tevatron, a similar collider experiment in the USA operating at lower-energies, found direct evidence for the Higgs boson.
- The Higgs boson was observed several datasets collected at the LHC
- The LHC (typically) publishes findings regardless of their statistical significance, i.e., significant results are not selectively reported.
- The LHC teams are guided by statistical committees, hopefully ensuring good practice
- The LHC is in principle committed to open data, which means a lot of the data should at some point become public. This is one recommendation for helping the crisis in social sciences.
- Typical training for experimentalists at the LHC includes basic statistics (although in my experience LHC experimentalits are still subject to the same traps and misinterpretations as everyone else).
- All members (thousands) of the experimental teams are authors on the papers. The incentive for bad practices such as $p$-hacking is presumably slightly lowered, as you cannot 'discover' a new effect and publish it only under your own name, and have improved job/grant prospects. This incentive might be a factor in the replication crisis in social sciences.
- All papers are subject to internal review (which I understand to be quite rigorous) as well as external review by a journal
- LHC analyses are often (I'm not sure who plans or decides this) blinded. This means that the experimentalists cannot tweak the analyses depending on the result. They are 'blind' to the result, make their choices, then unblind it only at the end. This should help prevent $p$-hacking
If anything, there is a suspicion that the practices at the LHC might even result in the opposite of the 'replication crisis;' analyses that find effects that are somewhat significant might be examined and tweaked until they decrease.
edited 5 mins ago
answered 11 mins ago
innisfreeinnisfree
11.5k32961
11.5k32961
add a comment |
add a comment |
$begingroup$
This is an answer to the question above, as I do not have access to the paper I cannot really discuss it.
Here is a simpler example: Take an Olympics athlete. How many failures before breaking the record? Is the record not broken because there may have been a thousand failures before breaking it?
What about the hundreds of athletes who try to reproduce and get a better record? Should they not try?
The statistics of failed experiments is similar: There is a goal (actually thousands of goals depending on the physics discipline), and a number of trials to reach the goal.
We know the LHC results are robust because there are two major and many smaller experiments trying for the same goals. The reason there are two experiments is so that systematic errors in one will not give spurious results. We trust that the measurement statistics that give the end results are correct, as we trust for the record breaking run that the measured times and distances are correct.
(And LHC is not an experiment. It is where experiments can be carried out depending on the efforts and ingenuity of researchers, it is the field where the Olympics takes place.)
The robustness of scientific results depends on the specific experimental measurements, not on integrating over all disparate experiments ever made. Bad use of statistics in mixing up apples and oranges and cauliflower and then applying it to the stock market.
Another way to look at it: If there were no failures , would the experiments mean anything? They would be predictable by pen and paper.
$endgroup$
2
$begingroup$
I'm not sure I buy the Olympics analogy. Failed attempts at breaking a record isn't the same thing as a failed attempt to reproduce an experiment. It also sounds like you are saying we should just cherry pick what does work and ignore when it fails.
$endgroup$
– Aaron Stevens
57 mins ago
$begingroup$
@AaronStevens It is the same as failed atempts for another runner to reproduce the record. I will edit, thanks
$endgroup$
– anna v
46 mins ago
$begingroup$
@AaronStevens " cherry pick what does work" but is not that evolution in general? and "ignore when it fails" one learns from failure to design better experiments.
$endgroup$
– anna v
37 mins ago
$begingroup$
and should you always expect no errors both in instruments and logic from experimenters? Science started with trial and error and evolved..Failures fall on the side, if not reproducible.
$endgroup$
– anna v
35 mins ago
1
$begingroup$
You're just making it sound like if just try hard enough we can reproduce anything (break the record) , but this is not always the case. Not being able to reproduce experiment A could mean that something was wrong with experiment A and that we shouldn't try to make too many predictions (or any at all) based on experiment A. We wouldn't want to keep chasing it (trying to break the record)
$endgroup$
– Aaron Stevens
23 mins ago
|
show 5 more comments
$begingroup$
This is an answer to the question above, as I do not have access to the paper I cannot really discuss it.
Here is a simpler example: Take an Olympics athlete. How many failures before breaking the record? Is the record not broken because there may have been a thousand failures before breaking it?
What about the hundreds of athletes who try to reproduce and get a better record? Should they not try?
The statistics of failed experiments is similar: There is a goal (actually thousands of goals depending on the physics discipline), and a number of trials to reach the goal.
We know the LHC results are robust because there are two major and many smaller experiments trying for the same goals. The reason there are two experiments is so that systematic errors in one will not give spurious results. We trust that the measurement statistics that give the end results are correct, as we trust for the record breaking run that the measured times and distances are correct.
(And LHC is not an experiment. It is where experiments can be carried out depending on the efforts and ingenuity of researchers, it is the field where the Olympics takes place.)
The robustness of scientific results depends on the specific experimental measurements, not on integrating over all disparate experiments ever made. Bad use of statistics in mixing up apples and oranges and cauliflower and then applying it to the stock market.
Another way to look at it: If there were no failures , would the experiments mean anything? They would be predictable by pen and paper.
$endgroup$
2
$begingroup$
I'm not sure I buy the Olympics analogy. Failed attempts at breaking a record isn't the same thing as a failed attempt to reproduce an experiment. It also sounds like you are saying we should just cherry pick what does work and ignore when it fails.
$endgroup$
– Aaron Stevens
57 mins ago
$begingroup$
@AaronStevens It is the same as failed atempts for another runner to reproduce the record. I will edit, thanks
$endgroup$
– anna v
46 mins ago
$begingroup$
@AaronStevens " cherry pick what does work" but is not that evolution in general? and "ignore when it fails" one learns from failure to design better experiments.
$endgroup$
– anna v
37 mins ago
$begingroup$
and should you always expect no errors both in instruments and logic from experimenters? Science started with trial and error and evolved..Failures fall on the side, if not reproducible.
$endgroup$
– anna v
35 mins ago
1
$begingroup$
You're just making it sound like if just try hard enough we can reproduce anything (break the record) , but this is not always the case. Not being able to reproduce experiment A could mean that something was wrong with experiment A and that we shouldn't try to make too many predictions (or any at all) based on experiment A. We wouldn't want to keep chasing it (trying to break the record)
$endgroup$
– Aaron Stevens
23 mins ago
|
show 5 more comments
$begingroup$
This is an answer to the question above, as I do not have access to the paper I cannot really discuss it.
Here is a simpler example: Take an Olympics athlete. How many failures before breaking the record? Is the record not broken because there may have been a thousand failures before breaking it?
What about the hundreds of athletes who try to reproduce and get a better record? Should they not try?
The statistics of failed experiments is similar: There is a goal (actually thousands of goals depending on the physics discipline), and a number of trials to reach the goal.
We know the LHC results are robust because there are two major and many smaller experiments trying for the same goals. The reason there are two experiments is so that systematic errors in one will not give spurious results. We trust that the measurement statistics that give the end results are correct, as we trust for the record breaking run that the measured times and distances are correct.
(And LHC is not an experiment. It is where experiments can be carried out depending on the efforts and ingenuity of researchers, it is the field where the Olympics takes place.)
The robustness of scientific results depends on the specific experimental measurements, not on integrating over all disparate experiments ever made. Bad use of statistics in mixing up apples and oranges and cauliflower and then applying it to the stock market.
Another way to look at it: If there were no failures , would the experiments mean anything? They would be predictable by pen and paper.
$endgroup$
This is an answer to the question above, as I do not have access to the paper I cannot really discuss it.
Here is a simpler example: Take an Olympics athlete. How many failures before breaking the record? Is the record not broken because there may have been a thousand failures before breaking it?
What about the hundreds of athletes who try to reproduce and get a better record? Should they not try?
The statistics of failed experiments is similar: There is a goal (actually thousands of goals depending on the physics discipline), and a number of trials to reach the goal.
We know the LHC results are robust because there are two major and many smaller experiments trying for the same goals. The reason there are two experiments is so that systematic errors in one will not give spurious results. We trust that the measurement statistics that give the end results are correct, as we trust for the record breaking run that the measured times and distances are correct.
(And LHC is not an experiment. It is where experiments can be carried out depending on the efforts and ingenuity of researchers, it is the field where the Olympics takes place.)
The robustness of scientific results depends on the specific experimental measurements, not on integrating over all disparate experiments ever made. Bad use of statistics in mixing up apples and oranges and cauliflower and then applying it to the stock market.
Another way to look at it: If there were no failures , would the experiments mean anything? They would be predictable by pen and paper.
edited 29 mins ago
answered 1 hour ago
anna vanna v
161k8153451
161k8153451
2
$begingroup$
I'm not sure I buy the Olympics analogy. Failed attempts at breaking a record isn't the same thing as a failed attempt to reproduce an experiment. It also sounds like you are saying we should just cherry pick what does work and ignore when it fails.
$endgroup$
– Aaron Stevens
57 mins ago
$begingroup$
@AaronStevens It is the same as failed atempts for another runner to reproduce the record. I will edit, thanks
$endgroup$
– anna v
46 mins ago
$begingroup$
@AaronStevens " cherry pick what does work" but is not that evolution in general? and "ignore when it fails" one learns from failure to design better experiments.
$endgroup$
– anna v
37 mins ago
$begingroup$
and should you always expect no errors both in instruments and logic from experimenters? Science started with trial and error and evolved..Failures fall on the side, if not reproducible.
$endgroup$
– anna v
35 mins ago
1
$begingroup$
You're just making it sound like if just try hard enough we can reproduce anything (break the record) , but this is not always the case. Not being able to reproduce experiment A could mean that something was wrong with experiment A and that we shouldn't try to make too many predictions (or any at all) based on experiment A. We wouldn't want to keep chasing it (trying to break the record)
$endgroup$
– Aaron Stevens
23 mins ago
|
show 5 more comments
2
$begingroup$
I'm not sure I buy the Olympics analogy. Failed attempts at breaking a record isn't the same thing as a failed attempt to reproduce an experiment. It also sounds like you are saying we should just cherry pick what does work and ignore when it fails.
$endgroup$
– Aaron Stevens
57 mins ago
$begingroup$
@AaronStevens It is the same as failed atempts for another runner to reproduce the record. I will edit, thanks
$endgroup$
– anna v
46 mins ago
$begingroup$
@AaronStevens " cherry pick what does work" but is not that evolution in general? and "ignore when it fails" one learns from failure to design better experiments.
$endgroup$
– anna v
37 mins ago
$begingroup$
and should you always expect no errors both in instruments and logic from experimenters? Science started with trial and error and evolved..Failures fall on the side, if not reproducible.
$endgroup$
– anna v
35 mins ago
1
$begingroup$
You're just making it sound like if just try hard enough we can reproduce anything (break the record) , but this is not always the case. Not being able to reproduce experiment A could mean that something was wrong with experiment A and that we shouldn't try to make too many predictions (or any at all) based on experiment A. We wouldn't want to keep chasing it (trying to break the record)
$endgroup$
– Aaron Stevens
23 mins ago
2
2
$begingroup$
I'm not sure I buy the Olympics analogy. Failed attempts at breaking a record isn't the same thing as a failed attempt to reproduce an experiment. It also sounds like you are saying we should just cherry pick what does work and ignore when it fails.
$endgroup$
– Aaron Stevens
57 mins ago
$begingroup$
I'm not sure I buy the Olympics analogy. Failed attempts at breaking a record isn't the same thing as a failed attempt to reproduce an experiment. It also sounds like you are saying we should just cherry pick what does work and ignore when it fails.
$endgroup$
– Aaron Stevens
57 mins ago
$begingroup$
@AaronStevens It is the same as failed atempts for another runner to reproduce the record. I will edit, thanks
$endgroup$
– anna v
46 mins ago
$begingroup$
@AaronStevens It is the same as failed atempts for another runner to reproduce the record. I will edit, thanks
$endgroup$
– anna v
46 mins ago
$begingroup$
@AaronStevens " cherry pick what does work" but is not that evolution in general? and "ignore when it fails" one learns from failure to design better experiments.
$endgroup$
– anna v
37 mins ago
$begingroup$
@AaronStevens " cherry pick what does work" but is not that evolution in general? and "ignore when it fails" one learns from failure to design better experiments.
$endgroup$
– anna v
37 mins ago
$begingroup$
and should you always expect no errors both in instruments and logic from experimenters? Science started with trial and error and evolved..Failures fall on the side, if not reproducible.
$endgroup$
– anna v
35 mins ago
$begingroup$
and should you always expect no errors both in instruments and logic from experimenters? Science started with trial and error and evolved..Failures fall on the side, if not reproducible.
$endgroup$
– anna v
35 mins ago
1
1
$begingroup$
You're just making it sound like if just try hard enough we can reproduce anything (break the record) , but this is not always the case. Not being able to reproduce experiment A could mean that something was wrong with experiment A and that we shouldn't try to make too many predictions (or any at all) based on experiment A. We wouldn't want to keep chasing it (trying to break the record)
$endgroup$
– Aaron Stevens
23 mins ago
$begingroup$
You're just making it sound like if just try hard enough we can reproduce anything (break the record) , but this is not always the case. Not being able to reproduce experiment A could mean that something was wrong with experiment A and that we shouldn't try to make too many predictions (or any at all) based on experiment A. We wouldn't want to keep chasing it (trying to break the record)
$endgroup$
– Aaron Stevens
23 mins ago
|
show 5 more comments
Thanks for contributing an answer to Physics Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fphysics.stackexchange.com%2fquestions%2f468891%2fhow-do-we-know-the-lhc-results-are-robust%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
$begingroup$
I think it is worth pointing out that the LHC doesn't just do one particle collision and then say the experiment is completed. How much do you know about what goes into such experiments, how many times they are actually repeated, and then how the data is analyzed from there?
$endgroup$
– Aaron Stevens
1 hour ago
$begingroup$
@AaronStevens I know some of it, but am not an expert. I know the LHC crashes two protons into each other multiple times, and the results of each collision are expected to be different but have different probabilities. Many of the daughter particles are unstable and also expected to decay. The detector sees the "final" products when they reach the detector, and the analysis is supposed to infer based on these detected particles what the original particles are. Does that answer your question?
$endgroup$
– Allure
56 mins ago
$begingroup$
I was asking if you had looked into the efforts taken to make sure the results from the LHC are good results and not just mistakes. Also the LHC isn't the only particle collider in existence.
$endgroup$
– Aaron Stevens
53 mins ago
$begingroup$
Related video
$endgroup$
– Aaron Stevens
42 mins ago
$begingroup$
For more about this important question than you may have bargained for, look for discussions about the "look-elsewhere effect" in the statistical analysis of the data from the mostly-independent ATLAS and CMS experiments at the LHC, especially in the context of their joint discovery of the Higgs particle.
$endgroup$
– rob♦
19 mins ago