1. Test reliability and validity
When you consider a test’s psychometric quality, then you have to consider the reliability and the validity of the test. The random measure errors is reflected in a test’s reliability.
For example, will the person get the same score independently of the time of testing, or are the person’s scoring too random to predict anything about the person’s personality or IQ?
Furthermore, you have to examine the not-random measure error, which is a test’s validity (Carmines & Zeller, 1979), i.e. is the test measuring exactly what it is supposed to measure?
Here you consider, whether the test’s items investigate what it is thought to consider. For example, if an item measures a personality trait that you don’t expect it to measure, then the test has a low validity.
2. Item analysis
You can calculate the inter-reliability of the different items through an item analysis so that the psychological test doesn’t have two items that say something about the same trait.
In this way, it will discriminate between low and high test values, which makes a normal distribution of test results, which is beneficial when you compare the test results (Coaley, 2010).
3. Factor analysis
A test can be compressed by making a factor analysis. A factor analysis is the compression of a number of items so that you end up with as few items as possible. In this way you only examine the values that have the most relevance for the specific issue of interest (Coaley, 2010). Thereby, one is able to improve the quality of the test.
4. Publication bias
At last, you need to be aware of the publication bias, sometimes called the file drawer problem, which is the tendency for a study to only pick out the positive/verifying results of a study.
At the same time, the study avoids to publish the negative falsifying results, and this will make a bias of reality, which the reader needs to have in mind when reading research literature because sometimes research has a hidden agenda, e.g. a political or economical agenda.
Sometimes, the interest or agenda of the study is written explicitly, but more often you need to consider the methodology of the study in order to see how the authors measure what they say they are measuring.
For example, the sample size, the sample characteristics, and the cross-sectional and the longitudinal studies (retrospective and prospective studies) are just a few of the factors that can influence a study’s results.
_____
Please note that I have only mentioned a few aspects of the psychometric quality. I am thinking about updating this article soon so that more aspects of testing are taken into account.
