Q: How can I evaluate qualitative data from different sources consistently and stringently without becoming too subjective and making too many assumptions?
For a research project, I have to evaluate and categorize several theoretical and practical Industry 4.0 use cases. Furthermore, I have to quantify the impact of each technology on 20 different competitive priorities. As each of the 500 use cases is written by a different author, and therefore, the size and quality of the text content varies greatly, it is almost impossible to evaluate the texts in a consistent way.
Thanks for your question. You have a relevant point. However, this would need the brains of our data/stats experts. We shall refer this to them, and get back to you thereafter.
Here are a few ways in which you can proceed. Note that these are relevant for evaluating quali data in general. We haven’t considered Industry 4.0 cases in particular.
- Assess the background and qualifications of the person providing the qualitative data. Does the person have the necessary knowledge, qualifications, or experience to make those judgments? Or has the person had that experience long enough to provide reliable data?
- Check whether the same data have been reported by several persons who are unlikely to be connected to each other. If various unconnected people are reporting the same data, it is likely to carry greater credence.
- Interview people providing the data – to form an independent judgment about the veracity of the data.
- Check for patterns – are the same types of data being reported from a specific geography/demographic group, etc. – and analyze factors that may be responsible for these patterns. By doing so, you can isolate data that are outliers to these patterns and further probe/verify inconsistent data.
- Try to create test situations using the qualitative data, and verify whether the results of these tests are predictable/as expected.
- Check if there is any precedent that supports the data.
- If possible, hire an independent agency to collect data and check if it corroborates the data you already have.
Hope that helps. And all the best for your project!
In addition, you may find it helpful to go through these related queries by other researchers: