Last issue of ACM SIGSOFT Software Engineering Notes (valume 37, issue 5), Panos Louridas and Georgios Gousios published a very interesting paper: “A Note on Rigour and Replicability”. They selected a research paper from ICSE’2012 as a counter-example of how to make a research experiment and publish it ready for replications. They analyzed such a paper and commented on every single point of lack of information about the reported experiment. In summary, that paper have been published in a famous Software Engineering conference with few information regarding to the experiment, which makes impossible a further replication.
In research context, without experiment replications, especially in different contexts, results’ generalization and knowledge building become hard (not to say that it is almost impossible). Moreover, with this lack of information on the experiment, it is even hard to compare results among similar experiments. In my Ph.D. research I need to analyze several research papers on mining software repositories, specially Change Request repositories (i.e., Bugzilla and Mantis). I have to admit that most of the papers I’ve read suffer with this issue. I can say that in none of them it is possible to replicate the findings or reuse the method for other context. Of course, it is necessary to give a discount to research field on Software Engineering since it is not mature as social sciences, for instance. Actually, we need to learn with these other more mature sciences.
On the other hand, Louridas and Gousios pointed to what I believe to be the right way to overcome this situation. They argued that Software Engineering conferences should require all necessary data and information for the experiment replication of accepted papers (or under review), such as: measurement data, interviews, questionnaires, research protocols, scripts of statistical programs (i.e., R or SPSS), and so on. Additionally, I believe that in first place it is also necessary to ensure that the reviewers and researchers are aware of a common protocol that should be followed for writing an experimental research paper. That is, both of them should ask to themselves: Does the paper provide enough information for the experiment replication? If all of these points were considered, then we can expect a significant advance on Software Engineering research.
Finally, for those who want to improve their experiment papers I suggest to take a look in the references pointed in . But do not be restricted to Software Engineering authors, also look for papers and books from social sciences research. Although medicine research is also a good source of information, in the context of experiments I believe that we are more close to the social sciences than we think. Indeed, apart the technology aspects of Software Engineering, it has a lot to do with human behavior. Thus, as a start point I would suggest the free book A Guide to Quantitative and Qualitative Dissertation Research .
 Louridas, P., & Gousios, G. (2012). A note on rigour and replicability. ACM SIGSOFT Software Engineering Notes, 37(5), 1. doi:10.1145/2347696.2347706
 Jr, J. S. (2012). A guide to quantitative and qualitative dissertation research. Retrieved from http://diginole.lib.fsu.edu/edpsy_faculty_publications/1/