principleMinor
Ideal number of participants for evaluation of development approach / software architecture
Viewed 0 times
idealnumberevaluationarchitectureparticipantsdevelopmentforsoftwareapproach
Problem
For my thesis, I have created a development approach (consisting of schemes, an application template and prototype code) that should make it easier to develop and adapt applications for a specific platform.
I have conducted interviews with developers of that platform to evaluate my approach. There have been 5 interviewees so far and the feedback that I got from them is sufficient for my scope, I would say.
However, in my thesis I would like to justify my decision that 5 evaluators are enough. Is there any scientific research or paper that suggests a certain number of evaluators for such a rather theoretical approach?
I know Nielsen's work about heuristic usability evaluation which says that with 5 participants, you discover around 75% of all problems in software usability / UI testing.
But as I said, my approach is no specific software product that has to be evaluated, but only the model itself, which is why I'm looking for research in that area.
I have conducted interviews with developers of that platform to evaluate my approach. There have been 5 interviewees so far and the feedback that I got from them is sufficient for my scope, I would say.
However, in my thesis I would like to justify my decision that 5 evaluators are enough. Is there any scientific research or paper that suggests a certain number of evaluators for such a rather theoretical approach?
I know Nielsen's work about heuristic usability evaluation which says that with 5 participants, you discover around 75% of all problems in software usability / UI testing.
But as I said, my approach is no specific software product that has to be evaluated, but only the model itself, which is why I'm looking for research in that area.
Solution
"Is there any scientific research or paper that suggests a certain number of evaluators for such a rather theoretical approach?"
There is a statistical method to calculate the correct sample size for an experiment.
This method is called a "power analysis" [1]. It calculates the minimum sample size needed to verify that an experiment's results are statistically significant. This approach assumes that you have at least two groups. A control group and an experimental group (i.e. control group uses the platform without your modifications, experimental group uses the platform with your modifications).
Thus, while there is a scientific process to calculate the correct sample size this approach is probably overkill for your study. Nielsen [2] states that heuristic evaluation is just a "cheap and quick" method for easy evaluation. As such, I believe you can justify a sample size of 5 by citing Nielsen. However, if you do want to standardize your evaluation process and use a statistical approach I encourage you to look into power analysis.
You may be interested in this online tool for calculating power.
There is a statistical method to calculate the correct sample size for an experiment.
This method is called a "power analysis" [1]. It calculates the minimum sample size needed to verify that an experiment's results are statistically significant. This approach assumes that you have at least two groups. A control group and an experimental group (i.e. control group uses the platform without your modifications, experimental group uses the platform with your modifications).
Thus, while there is a scientific process to calculate the correct sample size this approach is probably overkill for your study. Nielsen [2] states that heuristic evaluation is just a "cheap and quick" method for easy evaluation. As such, I believe you can justify a sample size of 5 by citing Nielsen. However, if you do want to standardize your evaluation process and use a statistical approach I encourage you to look into power analysis.
You may be interested in this online tool for calculating power.
- http://en.wikipedia.org/wiki/Statistical_power
- http://www.nngroup.com/topic/heuristic-evaluation/
Context
StackExchange Computer Science Q#12378, answer score: 5
Revisions (0)
No revisions yet.