Analysis can have different meanings to different people and therefore we distinguish different levels of analysis.
Basic Level Analysis
If a test with known psychometric properties is used individual tests takers, small groups and large cohorts can be scored.
It is assumed that the test has the desired psychometric properties to be used for scoring candidates. For each candidate a score is determined with or without weightings of questions. Descriptive statistics such as the mean, median, standard deviation, minimum, maximum, skewness and kurtosis are reported in tables as well as graphically. If required, different listings of the results of candidates can be compiled – for example, merit order, alphabetical order of ID order. If a pass mark is known, the pass rate and successful candidates can be reported. Individual test taker reports, as well as cohort reports, can be generated. Interpretation of scores for inferences can be provided from norm tables, as percentiles and/or descriptive.
Intermediate Level Analysis
The scoring of test takers and feedback provided at the basic level is enhanced through thorough analysis of the test itself. Classical item (question) statistics such as each item’s difficulty and discrimination index (point biserial) and test statistics such as reliability and the standard error of or measurement, is reported. For inferential purposes the analysis is done at the interval level in the framework of Rasch Measurement or an appropriate Item Response Theory (IRT) model. Once the targeting of the questions to the test takers candidates has been investigated, the abilities of the cohort are expressed on the same scale as the difficulties of the questions which provide achievement as well as diagnostic information. If subscales are defined, performance in each scale is reported. Real differences between scores are determined to refine scores (measures). For example, it requires more ability to improve from 48 to 49 on a 50-question test than to improve from 20 to 21 since the person is confronted with the most difficult questions in the former. In addition to feedback on test taker person performance, detailed item analysis is provided. These analyses include different question formats such as MCQs, EMQs, SMQs, essay type questions, etc.
Advanced Level Analysis
The intermediate level analysis is provided, but further detailed analyses explore the dimensionality of the test, investigate differential item functioning and possible bias using item and test information functions, test functioning construction and other required psychometric properties.