Q: How can I respond to reviewer comments regarding the sample size of my pilot study?

Detailed Question -

Since there were no previous studies to refer to, we conducted a study under the name "Pilot study" and contributed to the protocol paper.

I finally got the answer after a very long review period, but fortunately I was asked to do a minor revision ... I would like to ask about the ambiguity.

Reviewers' comments 1 : 

"Sample size: The stated sample size of 50 is larger than what would be considered standard for a pilot study. Given the lack of certainty regarding the benefit of the proposed technique, it may be reasonable to reduce this enrollment goal if the desire is only to show feasibility in methods and to obtain an estimate of effect size"

I thought that the number of samples could be arbitrarily set in the case of a pilot study. It would be okay if it was not too big and too small. But the reviewers think that our sample size is big. If you search for the number of samples in the preliminary study, there are many different opinion. (10 to 30 people?)

Reviewers' comments 2 : 

"Data analysis: Since the authors have declared this a pilot study and no sample size or power calculation has been performed on their primary outcome measure, they should include a sentence in their analysis plan that all of their comparisons will be considered exploratory in nature regardless of the significance level found. "


Since it is a pilot study, I did not do sample counting. However, I cannot understand below reviewers' comments. I do not know what this means.

"They should include a sentence in their analysis plan that all of their comparisons will be considered exploratory in nature, regardless of the significance level found."

May I ask you what would be a good response?

1 Answer to this question
Answer:

Unfortunately, I will not be able to give any clear suggestions on reviewers’ comments related to revising a manuscript since I have not perused the manuscript in its entirety. It would be best if you consult your supervisors and colleagues for a definitive answer to this. I can only give you some generic guidance on what the comments mean and how you can handle them:

 

As per Reviewer 1's comment, the authors need to justify the sample size. For this, you can refer to A tutorial on pilot studies: the what, why and how, which gives good details regarding sample size calculations for pilot studies. Also, GPower3 is a good software to calculate sample size.

 

Reviewer 2 is of the opinion that since no sample size has been determined at the start of the study, it may have an impact on the primary/secondary outcomes of the study; any statistical analyses used for calculation of the significance of the results obtained will not hold good since the initial sample size has not been calculated. Hence, the reviewer suggests that you should include a statement in the manuscript saying that the significance you have found relating to your outcome measurement (efficacy of the protocol or proposed technique) and the comparisons thereof is exploratory and not confirmative. I am assuming that the hypothesis and the level of significance were not established a priori and hence the reviewer is suggesting this.

 

Related reading:

Comment