Quantcast
Viewing latest article 4
Browse Latest Browse All 7

Re: Psychology must learn a lesson from fraud case

I agree with the author's sentiments regarding need for more sharing and more openness. However, I suspect that reluctance to share raw data stems from the fact that there are often multiple acceptable ways of analyzing data, and all of them may not lead to the same result. One would not worry at all about sending the data to a coauthor or a known colleague, or to a neutral party. However, many would think twice before sending it so someone who has a vested interest in proving you wrong. Often, people requesting data are these people and not neutral parties.

Suppose someone has spent 20 years of his career arguing for a position. Suppose Kim de Jong does a great study that shows that that position is wrong, and some other effect exists. Kim's effect is at p < 0.02. Now Kim sends uncleaned raw data to the adversary. The adversary uses different, but acceptable methods to clean data and to remove outliers. There are many valid ways to removing outliers, and they have different degrees of power and sensitivity. The adversary picks one or two methods lead to the worst results, and now the effect is at p < 0.07, no longer significant. Both argue back and forth, and what Kim has done is fine and there is no need to retract the paper. However, a seed do doubt is planted, and possibly Kim's reputation is tarnished somewhat when the adversary tells people that he looked at the data and the effect wasn't really there or was too weak to be believable.

One might say that 'if the effect is real, it should withstand any analysis'. This is flatly not true. Different methods have different amounts of power and subjective judgements need to be made (e.g., Windsorizing data to remove outliers is not very powerful, but is used and completely acceptable). There are generally more ways of producing a negative results than a positive one. If your goal is to produce a negative result, it will be possible in many cases if you try hard enough. True, if the effect is at p < 0.00001, it would still be significant under any good analysis. But in the real world, you don't always find such strong effects due to practical limitations. For example, you could only get so many patients of a certain type to participate in the study. You are not able keep adding patients, or have them do a lot of trials, to get a very strong effect. So you have to live with effects that are true but are not super strong. If someone is desperate to discredit you, they can get some mileage by trying all kinds of things with raw data.

This may be the reason why some are reluctant to share data easily, and not necessarily malice or secrecy.


Viewing latest article 4
Browse Latest Browse All 7

Trending Articles