Q: Should I reply to the editor for deciding that my paper (that was actually an expansion of my previous paper) was 'self-plagiarized'?

We are very sorry to know that your manuscript was rejected, and we understand that this is an especially difficult decision to come to terms with because the reason was self-plagiarism. Trying to explain/justify the level of overlap between your previous paper and the recent one might come across as being defensive to a journal, so perhaps that may not be the best course of action. We advise you to consider the following points before you decide to communicate with the journal:

• Simply citing a previous paper may not help avoid the problem of plagiarism (including self-plagiarism). It is always best to rewrite or paraphrase the text adequately. If the extent of text overlap between your previous paper and the current one was high, this may be considered plagiarism even if you had cited your previous paper. Moreover, self-plagiarism may also be a case of copyright infringement if the copyright of your previous article rests with the journal in which it was published and you have used considerable chunks of text from that paper. This is an important aspect that journals will take into account.
• The sections where the text overlap occurs matters. The materials and methods section tends to be the section with greater chances of text overlap if an author has used the same methods as in a previous study and if there are limited ways of describing complex technical details. Sometimes, even the study background or literature review section may overlap slightly with the corresponding section from a previous paper written by the same author, if the areas of research are similar. Journals may be a little accommodating if these are the sections where the texts overlap. However, if the overlap occurs in sections that describe the study objective, hypothesis, results, discussion, or conclusion, journals are typically far stricter.
• Critically review your manuscript to see how different it is from your previous paper. You mention that your current manuscript was an expansion of/improvement over your previous one. The question of text similarity aside, how different was your manuscript from the previous paper at the concept level? Would it qualify as a new study with research objectives/findings distinct enough from those of your previous study? What was it about this manuscript that would merit its publication as a new, unique contribution to your field of work?
• Understand journal requirements/policies with respect to ethical publication. With respect to issues like plagiarism/text recycling (and related concepts such as salami slicing), journals may differ slightly in their views of what is acceptable and what is not. Perhaps it would help to review the guidelines/policies of the journal to which you submitted your article, to understand why your manuscript was rejected on grounds of plagiarism, if you have not already done so. Alternatively, review again any details related to this aspect that the journal editor may have shared with you in the decision letter.
• Plan your next steps carefully. While this decision may be a setback to your publication plan, it may also be an important opportunity to learn more about a complex issue such as self-plagiarism and how to avoid it. You could redo/rewrite your article based on your learning and decide where to submit it next. If you prefer to submit your revised manuscript to the same journal, perhaps you could write to the editor thanking them for their feedback and ask if you may submit a revised version as a fresh submission. It will be important to ensure that you do not appear defensive about the journal’s decision and to communicate an eagerness to accept the feedback, understand journal requirements, and work towards making your article ready for submission.

Wish you all the best!

Additional resources:

• What's the big deal about self-plagiarism?
• Two of my review papers have similar Methods and Discussion sections. Is this self-plagiarism or salami slicing?
• Is it self-plagiarism to use the introduction of my published paper for another manuscript?
• How to avoid plagiarism and save your research career [Course]