Comment Social science peer review is a joke (Score 1) 97
I'm a bit salty since I spent all morning peer reviewing a paper. I don't know why; I get no recognition, credit, or pay for it. I do it mostly out of duty to try to help people get better results.
But this approach won't work. The main reason is that data are proprietary. I have to trust that they did their analyses correctly. I could "request the data from the author", and then spend days trying to decipher their coding and then try to replicate their analyses. That is if they give the data. In many cases, data come from proprietary sources or they won't share since they want to derive more publications from the data. I understand this push; once I got tenure I shared all my data freely. But before then, I had to guard those datasets.
I don't know what it is like in the harder sciences, but at least in those cases a discovery should be reproducible. If I make X claim about some physics or chemistry phenomenon, it is useless if it can't be replicated. However, in social sciences (and medicine), I can claim that my dataset says X and without the data and the steps of the analyses, that's impossible to prove, even if they find the reverse.
There is no way to fix this without incentivising peer review and even then, the reviewers would have to be just as motivated to really investigate the data. Since there are thousands of papers published every day, there's no way this happens.