Loved this, David! A quote I heard from Jason Zweig WSJ, in his being gobsmacked about his conversations with Kahneman said something like ... "he has no sunk costs." If he's been working on something for a while and it turns out not to be the thing, or he has new information he has no problem giving up the ghost and moving on.
That's been incredibly helpful to me not just in business but in sports, relationships and in general living my life.
I want to be right. It's nice to be right. But I don't care about being right.
The merciless environment of investing, where information is noisy and stakes are high, heavily penalizes the sunk cost fallacy. The really good professional investors like Zweig learn this early. Or if they're preternaturally wise, as Ace Greenberg (Bear, Stearns) seems to have been, they learn it from others--in Greenberg's case, his father, a retailer who drummed this lesson into his son.
So not a surprising comment, considering the source, but still cool and unique.
Great piece David, it reminds me of a Richard Dawkins story: "A formative influence on my undergraduate self was the response of a respected elder statesmen of the Oxford Zoology Department when an American visitor had just publicly disproved his favourite theory. The old man strode to the front of the lecture hall, shook the American warmly by the hand and declared in ringing, emotional tones: ‘My dear fellow, I wish to thank you. I have been wrong these fifteen years.’ And we clapped our hands red. Can you imagine a Government Minister being cheered in the House of Commons for a similar admission? “Resign, Resign” is a much more likely response!"
This is AMAZING! Do you have a recommendation of where I can read this story? ...and the end, of course, is dispiriting. So-called "flip-flopping" is a hallmark of people with good judgment, and yet we punish it mercilessly in the public sector.
Well done. If I had written a book that was read by Daniel Kahneman, I would have a lot of trouble not getting a bit cocky and casually dropping that fact into every conversation I had. I also think that it says a lot about Daniel in that he didn't just, "stay in his lane, " as far as academics and research go. He seemed to be curious about everything. I also took some wisdom from him when he said/wrote that he was always happy to find out when he was wrong, so that he could change his mind.
Haha...well, of course, he could've been wrong about my book;) But seriously, I completely agree with you, particularly the "lane" comment. A voraciously curious person, and gentle and incisive at the same time. Thanks for this, Christopher.
This is excellent. I just wanted to share that I think we live in a society, generally speaking, that praises and values being right above everything else. So, it has become extremely difficult to find people who just accept and admit: "I was wrong about this". The fact that anyone is wrong about something, does not invalidate their work. It takes courage to admit, when you made a mistake or, you changed your mind with new information. I hope more people follow Daniel Kahneman's example and start accepting when something about their work was wrong or simpler could have been stronger. We need it in these complicated times.
I agree, Gonzalo, we need more role models in the public eye with the courage to change their mind and explain their rationale. As Jeff Bezos said, “Anybody who doesn’t change their mind a lot is dramatically underestimating the complexity of the world we live in.”
I think a lot of it is in the set up, so that you’re stating “this is my opinion based on what I currently know and I reserve the right to change my mind.” Or as Dr. Joe Dispenza often says, “here’s what I think in my current state of ignorance…”
Daniel Kahneman's research and together with your conversation with him, it is always a mindset that grows in intellect, in learning- all i can think of this is the art of subtle unlearning and relearning. That's the greatest thing- sometimes with logic, stats and sometimes something as profound as transcendental as love - with mankind and humanity. Our choices and reseach can be followed by the -throw a rock on the lake and see the ripple effect. The effect he left on us. Flip the coin in the air and deep down we would know what we ve to make a decision off. Love reading your newsletter.
Hey David, this is a beautiful tribute to a true legend. I read Thinking, Fast and Slow during the early days of the pandemic, I also read Range around this time, probably my 2 favourite books and definitely 2 that impacted my thinking more than any other. I've since read Noise which I also enjoyed, it's very different to Thinking, Fast and Slow but still a very interesting concept. That story is quite something! Prepping for lunch.. 'That's a thing' haha.. it must have been such an honour to have been contacted by the great man, especially as a young author, for someone of his standing to take an interest in your work is quite something! And that's a lovely memento to have! (the signed paper). I'd love to know did you speak to him about Range? I would bet that he would really have enjoyed it. I actually have had that thought before so it is great to know that you connected and stayed in touch. And as for the lesson, what a wonderful, humble message, if such a legend of the psychological field is willing to change his mind, then why wouldn't anyone be willing to. Loved this one David, thanks.
William, so glad you enjoyed this one, as I know from your previous comment that you appreciated Kahneman as much as I did. It was an indeed an honor to hear from him, and with my reporter hat on, I was hoping we'd have a nice lunch so that I could follow up and learn more from him. He was very easy to talk to, and asked a lot of questions, so we had a good time. I didn't really have Range in mind at that time, so didn't really talk about it. Actually, that's not entirely true. The general idea was in my head, but I wasn't yet thinking of it as a project, just a curiosity, so I don't think I brought it up. (I tend to be self-conscious about my ideas early on...and often later on too, really, but when people say: "Am I allowed to ask you what you're working on?", I theoretically love to share, because they may have tips, but sometimes I'm sheepish because my ideas can be painfully nebulous.) Later on when I followed up with him, though, we discussed some of his work that I felt was relevant. In any case, I think he was an all around good force, and will be sorely missed.
Hey David, thanks for your reply, I noticed there's a decent volume of comments now and I'm sure you're pretty busy!! That's a lovely connection to have made and thanks for sharing it. He will indeed be sorely missed, but also great that his wonderful mind was tip top right until the end. I hope that the book project is going well, and I think it's very understandable for your ideas to be nebulous, I can't even comprehend the hours and research involved in distilling them into book sized presentations, but please keep doing what you do because the results are more than worthwhile. Thanks, William.
I always appreciate our interactions, and will always try to reply. You may have to accept an increasing number of typos or grammar errors, though, as I don't read back over comments here like I do with the post itself.
Of course! That reminds me of a wonderful term I read recently in Adam Alter's book about 'satisficing' - that one really stuck with me. You can't be a perfectionist about everything or nothing will ever get done.
"Social priming results, it turned out, were too good to be true, and researchers simply weren’t publishing their failed results. It left a body of work that appeared consistently convincing."
At this point I was waiting for you to mention Abraham Wald and his survivorship bias observation at the Statistical Research Group. 😉
From time to time I've thought about doing a short post on Wald just because it's such a neat example. But then I think maybe too many readers of this newsletter will have heard it already. What do you think...what portion of readers would already know the example?
Heh, I think the percentage of your readers who know about Wald's bomber is obviously MUCH higher than among the general public, BUT I imagine it's less than 50%. And an interesting post (to me, anyway) could be a look at a handful of OTHER examples of survivorship bias. In sports, in business, in education, in healthcare. Plus, Wald is so damn cool and smart that I can hear it again and again, hahaha!
To elaborate on my 50% wild-ass guess above, I always try to think that any audience, even "the general public", is always evolving. A moving river, with people coming in and people leaving. "David's audience" is always evolving. As they say, "think about the children!", some of whom are arriving, wide-eyed, at the gates of Epsteinworld every day, and will be delighted to have the tour. ;)
You might be interested the "Real Science of Sport Podcast" from 3/13/24--Can We Trust Sport Science Research? Ross Tucker brings up both the "underpowered study" problem and the misleading tendency of sports science journals to favor articles with Click-bait headlines over negative results on the same topics.
Hey Megan, Ross is a friend (as in a stay-at-my-house friend), but I completely missed that, so thanks! Christie Aschwanden has done some fantastic work in that area as well. Regarding the click-bait point, I read a paper a few years ago that showed that the most prominent journals (particularly in medicine) were mostly likely to have retractions, apparently because they were increasingly going for the counterintuitive findings. So apparently click-bait is definitely a thing even in specialty journals.
This is so admirable. Every time you write about someone who has passed, I appreciate the personal and unconventional angles you take. Also, between Kahneman's passing and the news of the NAIA's ban, I was wondering if one or both of these would make its way into Range Widely. I have read so many people argue that this is the appropriate way to respond to mistakes, but it's so much more powerful to actually see someone do it. Especially someone who is successful enough that they could get away with ignoring everyone! Now this has me wondering, and I'm curious if you'll entertain me with a prediction or two. Is there anything you have a hunch that we might all soon learn that the science's current consensus is wrong?
Hey Matt, I've been mulling this over a bit. (I actually had completely missed the NAIA ban... I see a lot less news the closer my book deadline gets...) And I agree with your sentiment, that there's a lot more lip service to this sort of thing than actual prominent examples we can easily hold up. Pablo Torre's podcast episode about alpha-wolf research is a good example, and I know he has collected examples of people who change their minds very publicly, so perhaps we'll see some more of that at some point. In any case, a prediction: this probably isn't what you were thinking, but I think that there is unfortunately a tremendous amount of stuff going on in cancer research that will eventually fall by the wayside. I say that because the replication rates for certain types of cancer research have been abysmal, but because the replications are attempted in cooperation with pharmaceutical companies, they don't necessarily share what the study was that didn't replicate. (Ex: https://www.nature.com/articles/483531a) ...But let me list a few others that I think will be reversed in some manner: a lot of "nudging" work...not all of it, but I think some ominous signs are there, and a lot of it raises the "tiny intervention/large effect" flag. (Doesn't mean that never happens, but it should raise suspicion.) I think some of the implicit bias work may not hold up well in terms of predicting actual behavior, or behavior change. Stereotype threat is another that has been very newsworthy that I'd guess will eventually fall away. And to end my controversial/newsworthy ones here: I think some of the responses to presumed-traumatic events (like "critical incident stress debriefing") will eventually be shown to do more harm than good. In medicine, I think some orthopedic surgeries will go, as we continually learn that huge numbers of people, upon imaging, have some tear but it isn't actually causing their symptoms. I think routine physicals for asymptomatic adults, at least younger ones, will eventually disappear, as will various cancer screenings, at least at younger ages. (Screening programs already seem rarely, if ever, to improve overall survival of the screened population, but they can producing misleading stats because they pick up stuff that one would never need to know about.) There are tons of correlations of things with vitamin D, some of which may be real, but many of which I think are selection bias. (Recommendations lead to healthier people taking up a practice, which then makes the practice look healthier than it is — if you don't account for that.) I think the scientific basis of chiropractic (subluxations as cause of illness) has already been shown not really to exist. Or at least it hasn't been shown to exist. That said, I think many in the field accept this and have adapted and do valuable things. I think some old medical dogma, like that one can't build new bone after 30 will be increasingly accepted as untrue, and I think so-called "blue zones" may not hold up well to scrutiny. Actually, I may do a post on that. So these are some things that come to mind.
This is such a great list, thank you! I had a professor who felt similarly about nudging. The point on health screenings for asymptomatic individuals rings of something Christie Aschwanden would endorse. One of my favorite points that she made in her book is that the rise of medical/exercise trackers and tests is so exhaustive that the thing it is best at is to convince healthy people that they are unhealthy. I also remember a teammate who tore his meniscus. He learned from his doctor that a surprisingly high number of people have a meniscus tear but don't know it because they aren't affected by it. I also read somewhere (maybe in a Tyler Cowen blog post?) that blue zones have an alarmingly high number of people just overstating how old they are or misremembering their birthdays, and adjusting the data for that took out a lot of what made these communities outliers.
The rest of the things you mentioned here are genuinely new to me though, and I found it really interesting. Do you think there is a common mistake or two that leads scientific consensus to get things wrong? Maybe underreporting/underpublishing of negative results (like with the cancer research you mentioned) or the bias towards counterintuitive findings (I saw you mentioned this one in another comment)?
You are so well read for a young'un! And that is a great way to summarize one of Christie's points...I think I might see her tonight so I'll show her this if so;) Regarding blue zones, that's exactly what I was thinking of. This actually reminds me that I want to do a post on it, since there was a big Netflix series about blue zones secrets. Not that I don't think there's plenty to learn, but I recall perusing a study that showed something like an inverse relationship between typical lifespan and likelihood of having supercentenarian outliers, which prompted my more expressive eyebrow to rise about three feet. ...In terms of common mistakes, I think both the things you mention are at play. (And there are ways when a body of work has grown to suss out the unde-reporting of negative results. Have you heard of a funnel plot?) In general, I think the shift to evaluating researchers based on number of publications has been a major backfire. It had led to a) an undeniable uptick in fraud, and b) researchers trying to get many papers out of, say, a particular experiment, so they do all this data slicing and dicing to get positive results, and all sorts of different results. Combine that with powerful statistical computer programs that every scientist can use without ever having learned what it's really doing, and there's just an epic flood of false positive results. In the case of priming, I think it was the allure of counterintuitive results, but also very much this slicing and dicing issue, where if an initial hypothesis is not supported, the researcher starts HARKing (hypothesizing after the results are known) until they find positive results. I think I'm going to write about exactly that in the book I'm working on now.
Wait... there's an *inverse* relationship between typical lifespan and likelihood of supercentenarian outliers?? Even my non-scientifically-trained self knows enough to be skeptical of that. I hadn't heard of funnel plots, but I just looked them up. They seem important! I don't like that HARKing is common enough that we've now named it. Regarding what you said about powerful statistical computer programs, you mentioned something about in your Masters work that you had some statistical problem like this right? Something to the effect of looking back you now realize you did something you shouldn't have with data?
Re: Christie Aschwanden- Oh my gosh!! please tell her next time that I am impressed with her ability to convey so much scientific research in such digestable, entertaining ways. Should she choose to write one, I'll be first in line for her next book.
Ok I'm going to have to go back and read that study, because I'm not sure if it was that straightforward, but it was something laughable like that. I'm now committed to doing a post at some point and will find out for sure! ..Regarding my masters, yes absolutely. I had a big database with a bunch of variables, and when my initial hypothesis wasn't supported, I just searched for other correlations. This is not bad as an exploratory exercise to find a hypothesis to then test. But it is bad when it is presented as the initial hypothesis that was being tested. I was so unwitting about what I was doing that, at least, that I left a pretty transparent trail. But when papers are published that's often not the case. It was only some years later, reporting about poor scientific practices, that I realized some of what I was learning seemed like something I had done! In retrospect, it's a little nuts that I was rushed into learning really detailed material about a tiny niche before I understood the fundamental of experiments and inference. ...And I'll let Christie know!
Ha! Whatever the study turns out to say, I'm excited to read the post when the time comes.
I appreciate how reflective you are about being rushed into learning detailed material, and I'm glad you can look back on it well. I'm just a little worried you're not the only scientist who was rushed that way and that not everyone else has realized!
Love your piece and Kahneman's "lesson I have learned"! I'm currently on chapter 3 of his book. I guess I'll be going into chapter 4 with a different mindset. :) Btw, “representativeness” reminds me of "search images" in vision.
They are elements of a person's visual life experience stored in a "library." Ellis Loew (Cornell U) described "search images" as "made up of the concepts associated with all recognizable forms, surfaces, and objects"; so, e.g., one "sees" an apple because one's perception best matches "apple" in the library. This is from Loew's section (2.3) on Human Factors in Photographic Interpretation of chapter 2, Fundamentals of Photographic Interpretation (I was author/editor), of The Manual of Photographic Interpretation, 2nd edition (https://tinyurl.com/387c5fzx). The Manual was published in 1997; Loew could provide any updates on this.
Ah, another one of my intellectual heroes (along with Mihaly Csikszentmihalyi and Anders Erickson) that I won’t be able to meet in person. Thanks for giving us a chance to share your experience vicariously, David. If only we can become a “the lesson I have learned” kind of society.
Thanks Renita! Interesting you mention Anders, as I had a very generative relationship with him, largely based on disagreement;) But I think we both moved one another's needles on a few ideas, so I think of that relationship as something Kahneman would have liked.
Loved this, David! A quote I heard from Jason Zweig WSJ, in his being gobsmacked about his conversations with Kahneman said something like ... "he has no sunk costs." If he's been working on something for a while and it turns out not to be the thing, or he has new information he has no problem giving up the ghost and moving on.
That's been incredibly helpful to me not just in business but in sports, relationships and in general living my life.
I want to be right. It's nice to be right. But I don't care about being right.
Oh man, I love that: "he has no sunk costs." How is that for a cool and unique compliment! Thanks so much for sharing that, Kevin.
The merciless environment of investing, where information is noisy and stakes are high, heavily penalizes the sunk cost fallacy. The really good professional investors like Zweig learn this early. Or if they're preternaturally wise, as Ace Greenberg (Bear, Stearns) seems to have been, they learn it from others--in Greenberg's case, his father, a retailer who drummed this lesson into his son.
So not a surprising comment, considering the source, but still cool and unique.
And a beautiful post!
Great piece David, it reminds me of a Richard Dawkins story: "A formative influence on my undergraduate self was the response of a respected elder statesmen of the Oxford Zoology Department when an American visitor had just publicly disproved his favourite theory. The old man strode to the front of the lecture hall, shook the American warmly by the hand and declared in ringing, emotional tones: ‘My dear fellow, I wish to thank you. I have been wrong these fifteen years.’ And we clapped our hands red. Can you imagine a Government Minister being cheered in the House of Commons for a similar admission? “Resign, Resign” is a much more likely response!"
This is AMAZING! Do you have a recommendation of where I can read this story? ...and the end, of course, is dispiriting. So-called "flip-flopping" is a hallmark of people with good judgment, and yet we punish it mercilessly in the public sector.
I loved it also, I wrote about it here: https://www.linkedin.com/pulse/time-clap-your-hands-red-higher-you-grow-deeper-bow-aidan-mccullen/
But the Dawkins quote is from a talk "Science, Delusion and the Appetite for Wonder": https://youtu.be/8MGJtZr20nE?si=wU2I8Ld9WpR14dKe
Well done. If I had written a book that was read by Daniel Kahneman, I would have a lot of trouble not getting a bit cocky and casually dropping that fact into every conversation I had. I also think that it says a lot about Daniel in that he didn't just, "stay in his lane, " as far as academics and research go. He seemed to be curious about everything. I also took some wisdom from him when he said/wrote that he was always happy to find out when he was wrong, so that he could change his mind.
Haha...well, of course, he could've been wrong about my book;) But seriously, I completely agree with you, particularly the "lane" comment. A voraciously curious person, and gentle and incisive at the same time. Thanks for this, Christopher.
This is excellent. I just wanted to share that I think we live in a society, generally speaking, that praises and values being right above everything else. So, it has become extremely difficult to find people who just accept and admit: "I was wrong about this". The fact that anyone is wrong about something, does not invalidate their work. It takes courage to admit, when you made a mistake or, you changed your mind with new information. I hope more people follow Daniel Kahneman's example and start accepting when something about their work was wrong or simpler could have been stronger. We need it in these complicated times.
I agree, Gonzalo, we need more role models in the public eye with the courage to change their mind and explain their rationale. As Jeff Bezos said, “Anybody who doesn’t change their mind a lot is dramatically underestimating the complexity of the world we live in.”
I think a lot of it is in the set up, so that you’re stating “this is my opinion based on what I currently know and I reserve the right to change my mind.” Or as Dr. Joe Dispenza often says, “here’s what I think in my current state of ignorance…”
Daniel Kahneman's research and together with your conversation with him, it is always a mindset that grows in intellect, in learning- all i can think of this is the art of subtle unlearning and relearning. That's the greatest thing- sometimes with logic, stats and sometimes something as profound as transcendental as love - with mankind and humanity. Our choices and reseach can be followed by the -throw a rock on the lake and see the ripple effect. The effect he left on us. Flip the coin in the air and deep down we would know what we ve to make a decision off. Love reading your newsletter.
Absolutely beautiful.
Thank you for taking the time to write this up. What an example of maturity and a mindset I want to emulate.
Daniel, thanks so much for the kind words, especially the "time" part, as it always takes me longer than I expect. (Kahneman's "planning fallacy"!)
Hey David, this is a beautiful tribute to a true legend. I read Thinking, Fast and Slow during the early days of the pandemic, I also read Range around this time, probably my 2 favourite books and definitely 2 that impacted my thinking more than any other. I've since read Noise which I also enjoyed, it's very different to Thinking, Fast and Slow but still a very interesting concept. That story is quite something! Prepping for lunch.. 'That's a thing' haha.. it must have been such an honour to have been contacted by the great man, especially as a young author, for someone of his standing to take an interest in your work is quite something! And that's a lovely memento to have! (the signed paper). I'd love to know did you speak to him about Range? I would bet that he would really have enjoyed it. I actually have had that thought before so it is great to know that you connected and stayed in touch. And as for the lesson, what a wonderful, humble message, if such a legend of the psychological field is willing to change his mind, then why wouldn't anyone be willing to. Loved this one David, thanks.
William, so glad you enjoyed this one, as I know from your previous comment that you appreciated Kahneman as much as I did. It was an indeed an honor to hear from him, and with my reporter hat on, I was hoping we'd have a nice lunch so that I could follow up and learn more from him. He was very easy to talk to, and asked a lot of questions, so we had a good time. I didn't really have Range in mind at that time, so didn't really talk about it. Actually, that's not entirely true. The general idea was in my head, but I wasn't yet thinking of it as a project, just a curiosity, so I don't think I brought it up. (I tend to be self-conscious about my ideas early on...and often later on too, really, but when people say: "Am I allowed to ask you what you're working on?", I theoretically love to share, because they may have tips, but sometimes I'm sheepish because my ideas can be painfully nebulous.) Later on when I followed up with him, though, we discussed some of his work that I felt was relevant. In any case, I think he was an all around good force, and will be sorely missed.
Hey David, thanks for your reply, I noticed there's a decent volume of comments now and I'm sure you're pretty busy!! That's a lovely connection to have made and thanks for sharing it. He will indeed be sorely missed, but also great that his wonderful mind was tip top right until the end. I hope that the book project is going well, and I think it's very understandable for your ideas to be nebulous, I can't even comprehend the hours and research involved in distilling them into book sized presentations, but please keep doing what you do because the results are more than worthwhile. Thanks, William.
I always appreciate our interactions, and will always try to reply. You may have to accept an increasing number of typos or grammar errors, though, as I don't read back over comments here like I do with the post itself.
Of course! That reminds me of a wonderful term I read recently in Adam Alter's book about 'satisficing' - that one really stuck with me. You can't be a perfectionist about everything or nothing will ever get done.
"Social priming results, it turned out, were too good to be true, and researchers simply weren’t publishing their failed results. It left a body of work that appeared consistently convincing."
At this point I was waiting for you to mention Abraham Wald and his survivorship bias observation at the Statistical Research Group. 😉
From time to time I've thought about doing a short post on Wald just because it's such a neat example. But then I think maybe too many readers of this newsletter will have heard it already. What do you think...what portion of readers would already know the example?
Heh, I think the percentage of your readers who know about Wald's bomber is obviously MUCH higher than among the general public, BUT I imagine it's less than 50%. And an interesting post (to me, anyway) could be a look at a handful of OTHER examples of survivorship bias. In sports, in business, in education, in healthcare. Plus, Wald is so damn cool and smart that I can hear it again and again, hahaha!
To elaborate on my 50% wild-ass guess above, I always try to think that any audience, even "the general public", is always evolving. A moving river, with people coming in and people leaving. "David's audience" is always evolving. As they say, "think about the children!", some of whom are arriving, wide-eyed, at the gates of Epsteinworld every day, and will be delighted to have the tour. ;)
You might be interested the "Real Science of Sport Podcast" from 3/13/24--Can We Trust Sport Science Research? Ross Tucker brings up both the "underpowered study" problem and the misleading tendency of sports science journals to favor articles with Click-bait headlines over negative results on the same topics.
Hey Megan, Ross is a friend (as in a stay-at-my-house friend), but I completely missed that, so thanks! Christie Aschwanden has done some fantastic work in that area as well. Regarding the click-bait point, I read a paper a few years ago that showed that the most prominent journals (particularly in medicine) were mostly likely to have retractions, apparently because they were increasingly going for the counterintuitive findings. So apparently click-bait is definitely a thing even in specialty journals.
This is so admirable. Every time you write about someone who has passed, I appreciate the personal and unconventional angles you take. Also, between Kahneman's passing and the news of the NAIA's ban, I was wondering if one or both of these would make its way into Range Widely. I have read so many people argue that this is the appropriate way to respond to mistakes, but it's so much more powerful to actually see someone do it. Especially someone who is successful enough that they could get away with ignoring everyone! Now this has me wondering, and I'm curious if you'll entertain me with a prediction or two. Is there anything you have a hunch that we might all soon learn that the science's current consensus is wrong?
Hey Matt, I've been mulling this over a bit. (I actually had completely missed the NAIA ban... I see a lot less news the closer my book deadline gets...) And I agree with your sentiment, that there's a lot more lip service to this sort of thing than actual prominent examples we can easily hold up. Pablo Torre's podcast episode about alpha-wolf research is a good example, and I know he has collected examples of people who change their minds very publicly, so perhaps we'll see some more of that at some point. In any case, a prediction: this probably isn't what you were thinking, but I think that there is unfortunately a tremendous amount of stuff going on in cancer research that will eventually fall by the wayside. I say that because the replication rates for certain types of cancer research have been abysmal, but because the replications are attempted in cooperation with pharmaceutical companies, they don't necessarily share what the study was that didn't replicate. (Ex: https://www.nature.com/articles/483531a) ...But let me list a few others that I think will be reversed in some manner: a lot of "nudging" work...not all of it, but I think some ominous signs are there, and a lot of it raises the "tiny intervention/large effect" flag. (Doesn't mean that never happens, but it should raise suspicion.) I think some of the implicit bias work may not hold up well in terms of predicting actual behavior, or behavior change. Stereotype threat is another that has been very newsworthy that I'd guess will eventually fall away. And to end my controversial/newsworthy ones here: I think some of the responses to presumed-traumatic events (like "critical incident stress debriefing") will eventually be shown to do more harm than good. In medicine, I think some orthopedic surgeries will go, as we continually learn that huge numbers of people, upon imaging, have some tear but it isn't actually causing their symptoms. I think routine physicals for asymptomatic adults, at least younger ones, will eventually disappear, as will various cancer screenings, at least at younger ages. (Screening programs already seem rarely, if ever, to improve overall survival of the screened population, but they can producing misleading stats because they pick up stuff that one would never need to know about.) There are tons of correlations of things with vitamin D, some of which may be real, but many of which I think are selection bias. (Recommendations lead to healthier people taking up a practice, which then makes the practice look healthier than it is — if you don't account for that.) I think the scientific basis of chiropractic (subluxations as cause of illness) has already been shown not really to exist. Or at least it hasn't been shown to exist. That said, I think many in the field accept this and have adapted and do valuable things. I think some old medical dogma, like that one can't build new bone after 30 will be increasingly accepted as untrue, and I think so-called "blue zones" may not hold up well to scrutiny. Actually, I may do a post on that. So these are some things that come to mind.
This is such a great list, thank you! I had a professor who felt similarly about nudging. The point on health screenings for asymptomatic individuals rings of something Christie Aschwanden would endorse. One of my favorite points that she made in her book is that the rise of medical/exercise trackers and tests is so exhaustive that the thing it is best at is to convince healthy people that they are unhealthy. I also remember a teammate who tore his meniscus. He learned from his doctor that a surprisingly high number of people have a meniscus tear but don't know it because they aren't affected by it. I also read somewhere (maybe in a Tyler Cowen blog post?) that blue zones have an alarmingly high number of people just overstating how old they are or misremembering their birthdays, and adjusting the data for that took out a lot of what made these communities outliers.
The rest of the things you mentioned here are genuinely new to me though, and I found it really interesting. Do you think there is a common mistake or two that leads scientific consensus to get things wrong? Maybe underreporting/underpublishing of negative results (like with the cancer research you mentioned) or the bias towards counterintuitive findings (I saw you mentioned this one in another comment)?
Update: I had dinner with Christie tonight and showed her this and she was very impressed with you as a reader;)
You are so well read for a young'un! And that is a great way to summarize one of Christie's points...I think I might see her tonight so I'll show her this if so;) Regarding blue zones, that's exactly what I was thinking of. This actually reminds me that I want to do a post on it, since there was a big Netflix series about blue zones secrets. Not that I don't think there's plenty to learn, but I recall perusing a study that showed something like an inverse relationship between typical lifespan and likelihood of having supercentenarian outliers, which prompted my more expressive eyebrow to rise about three feet. ...In terms of common mistakes, I think both the things you mention are at play. (And there are ways when a body of work has grown to suss out the unde-reporting of negative results. Have you heard of a funnel plot?) In general, I think the shift to evaluating researchers based on number of publications has been a major backfire. It had led to a) an undeniable uptick in fraud, and b) researchers trying to get many papers out of, say, a particular experiment, so they do all this data slicing and dicing to get positive results, and all sorts of different results. Combine that with powerful statistical computer programs that every scientist can use without ever having learned what it's really doing, and there's just an epic flood of false positive results. In the case of priming, I think it was the allure of counterintuitive results, but also very much this slicing and dicing issue, where if an initial hypothesis is not supported, the researcher starts HARKing (hypothesizing after the results are known) until they find positive results. I think I'm going to write about exactly that in the book I'm working on now.
Wait... there's an *inverse* relationship between typical lifespan and likelihood of supercentenarian outliers?? Even my non-scientifically-trained self knows enough to be skeptical of that. I hadn't heard of funnel plots, but I just looked them up. They seem important! I don't like that HARKing is common enough that we've now named it. Regarding what you said about powerful statistical computer programs, you mentioned something about in your Masters work that you had some statistical problem like this right? Something to the effect of looking back you now realize you did something you shouldn't have with data?
Re: Christie Aschwanden- Oh my gosh!! please tell her next time that I am impressed with her ability to convey so much scientific research in such digestable, entertaining ways. Should she choose to write one, I'll be first in line for her next book.
Ok I'm going to have to go back and read that study, because I'm not sure if it was that straightforward, but it was something laughable like that. I'm now committed to doing a post at some point and will find out for sure! ..Regarding my masters, yes absolutely. I had a big database with a bunch of variables, and when my initial hypothesis wasn't supported, I just searched for other correlations. This is not bad as an exploratory exercise to find a hypothesis to then test. But it is bad when it is presented as the initial hypothesis that was being tested. I was so unwitting about what I was doing that, at least, that I left a pretty transparent trail. But when papers are published that's often not the case. It was only some years later, reporting about poor scientific practices, that I realized some of what I was learning seemed like something I had done! In retrospect, it's a little nuts that I was rushed into learning really detailed material about a tiny niche before I understood the fundamental of experiments and inference. ...And I'll let Christie know!
Ha! Whatever the study turns out to say, I'm excited to read the post when the time comes.
I appreciate how reflective you are about being rushed into learning detailed material, and I'm glad you can look back on it well. I'm just a little worried you're not the only scientist who was rushed that way and that not everyone else has realized!
Love your piece and Kahneman's "lesson I have learned"! I'm currently on chapter 3 of his book. I guess I'll be going into chapter 4 with a different mindset. :) Btw, “representativeness” reminds me of "search images" in vision.
Haha...got to you just in time;) The rest has held up well. ...What are "search images" in vision? I'm intrigued...
They are elements of a person's visual life experience stored in a "library." Ellis Loew (Cornell U) described "search images" as "made up of the concepts associated with all recognizable forms, surfaces, and objects"; so, e.g., one "sees" an apple because one's perception best matches "apple" in the library. This is from Loew's section (2.3) on Human Factors in Photographic Interpretation of chapter 2, Fundamentals of Photographic Interpretation (I was author/editor), of The Manual of Photographic Interpretation, 2nd edition (https://tinyurl.com/387c5fzx). The Manual was published in 1997; Loew could provide any updates on this.
Great piece David!
Thanks for reading, Ankur!
Thanks so much!! Excellent info update, and great food for thought.
Thanks so much for reading, Angela, and so glad you found it thought-provoking.
Amazing story, full of insights. I love how Daniel, “a man who changed how we think about thinking was also willing to change his own thinking.”
Thanks for the kind words, and for reading Bogdan!
Ah, another one of my intellectual heroes (along with Mihaly Csikszentmihalyi and Anders Erickson) that I won’t be able to meet in person. Thanks for giving us a chance to share your experience vicariously, David. If only we can become a “the lesson I have learned” kind of society.
Thanks Renita! Interesting you mention Anders, as I had a very generative relationship with him, largely based on disagreement;) But I think we both moved one another's needles on a few ideas, so I think of that relationship as something Kahneman would have liked.
Love this essay. Thanks for writing it!
Thanks so much for reading Will!