Valid and reliable instruments are needed for assessment and comparison of training outcomes after various methods of cardiopulmonary resuscitation training. Trials were retrieved by searching MEDLINE (1990–February 2005) and using the reference lists of original communications and reviews. Studies were considered relevant if they included an intervention, a study population of life support providers randomized and divided into groups and an evaluation or assessment of the performance. The studies were analyzed and scored to assess their validity. Twenty-five studies fulfilled the criteria. Nineteen of them assessed cardiopulmonary resuscitation skills, four cardiopulmonary resuscitation and defibrillation and two assessed defibrillation only. The mean number of participants was 107 (range 36–495). A wide variety of assessment methods were used in the studies with methodological shortcomings. Most studies in this review compared participants with each other, not against a standard or a defined passing level. Qualified studies with well defined study populations, standardized study settings and explicit, comparable outcomes would be needed to assess the quality of cardiopulmonary resuscitation and defibrillation performance.