Transcript
1/26/2014 CREOG Exam Scores – What do they really mean? Peter Hong, M.D. Residency Program Director Nassau University Medical Center East Meadow, N.Y. a class= __cf_email__ href= /cdn-cgi/l/email-protection data-cfemail= bbcbd3d4d5dcfbd5ced6d895dedfce [email protected] /a script data-cfhash='f9e31' type= text/javascript /* ![CDATA[ */!function(t,e,r,n,c,a,p){try{t=document.currentScript||function(){for(t=document.getElementsByTagName('script'),e=t.length;e--;)if(t[e].getAttribute('data-cfhash'))return t[e]}();if(t&&(c=t.previousSibling)){p=t.parentNode;if(a=c.getAttribute('data-cfemail')){for(e='',r='0x'+a.substr(0,2)|0,n=2;a.length-n;n+=2)e+='%'+('0'+('0x'+a.substr(n,2)^r).toString(16)).slice(-2);p.replaceChild(document.createTextNode(decodeURIComponent(e)),c)}p.removeChild(t)}}catch(u){}}()/* ]] */ /script Precis ã CREOG scores are under appreciated. Converting scores to percentiles allow for comparison of a resident’s/training program’s standing with their peers in areas of knowledge depth, progress of learning & risk of failing written Board exams. Intrepretation basis explained, tools provided. Learning Objectives ãAdvise residents of their test results in comparison with their peer residents nationally ãMonitor resident’s progress & national standing in learning over time ãEasily convert scores to percentiles ãAssess failing risk on written Board exam I have no conflicts of interest 1 1/26/2014 Administer the exam as instructed It’s the only meaningful opportunity to assess your residents knowledge foundation Major Test Categories & Breakdown 2012 ITEMS 2013 ITEMS General Considerations 29 (9%) 24 (8%) Primary and Preventive Ambulatory Health Care 50 (15%) 50 (16%) Obstetrics 75 (23%) 75 (24%) Gynecology 76 (23%) 74 (23%) Reproductive Endocrinology 42 (13%) 40 (13%) Oncology 37 (11%) 42(13%) Genomics 14 (4%) 13 (4%) 323 318 CREOG QUESTION CATEGORY TOTAL EXAMINATION Common Beliefs & Misconceptions ã Test Scores are linearly related to # questions answered correctly ã Similar scores in subsequent years will happen if I don’t read ã Occasional reading will significantly increase my CREOG scores ã Test scores provide little or unknown value in risk assessment towards failing ABOG written exam Each of the above points are fallacies 2 1/26/2014 From Exam to Score To CREOG for grading & analysis Exam Results Comparative 3 Digit Score Where to focus on the Report? There are about 1100 residents in training in each level Nationally Life Distributes in a Bell Shape Curve ã Stature – there are skinny people, there are obese people ã Height – there are short people, there are tall people ã IQ – there are morons, there are bright ã In each of the above, there are also average people which account for the majority 3 1/26/2014 Life Distributes in a Bell Shape Curve Stature Height IQ Skinny Short Moron Obese Tall Bright Average What about CREOG comparative 3 digit scores? CREOG Scores distributes in a bell shape curve also This test score is 200 These 550 Residents know less than the residents who scored at this line 50% of test takers scored BELOW this level 50% of test takers scored ABOVE this level These 550 Residents know more than the residents who scored at this line This test score is 180 16 % of test takers scored BELOW this line 84 % of test takers scored ABOVE this line These 924 Residents know more than the residents who scored above this line 176 Residents 4 1/26/2014 This test score is 160 2 % of test takers scored BELOW this line 98 % of test takers scored ABOVE this line These 1067 Residents know more than the residents who scored above this line 22 Residents Facts to Consider ã CREOG is the only standardize national exam administered to residents ã CREOG assesses resident education ã ABOG assesses ability to practice ã Both CREOG & ABOG ultimately tests knowledge ã Test are designed to be discriminators, (i.e. there are passers & failures (e.g. ABOG, USMLE, COMLEX) ã About 15% of ABOG test takers fail every year Counseling residents post CREOG Program B – given result printout, advised to read more Program A – given result printout, advised to do better next year Resident Program C – given result printout, did you know that your 160 places you in the lower 2% of test takers and that the lower 15% of ABOGers fail. How much medical school debt do you have? Remediate! 5 1/26/2014 Charting Resident CREOG Performance 0.025 0.025 Resident A PGY 4 0.02 0.02 0.015 0.015 Resident A PGY 3 Resident A PGY 2 0.01 0.01 0.005 0.005 0 0 140 160 180 200 220 240 260 Resident A PGY 1 Score %ile PGY 4 2013 CREOG 192 36 PGY 3 2012 CREOG 194 40 PGY 2 2011 CREOG 175 13 PGY 1 2010 CREOG 177 14 Normal Curve with 3 Digit Comparative Scores 0.025 0.025 0.02 0.02 0.015 0.015 0.01 0.01 0.005 0.005 0 0 140 160 180 200 220 240 260 National CREOG Distribution 15% Line National Mean Conversion of Comparison Score to Percentile *Partial List 6 1/26/2014 Does the Same Score Each Year Mean No Growth in Knowledge? Medical Knowledge PGY 4 PGY 3 PGY 2 PGY 1 Medical Student Does the Same Score Each Year Mean No Growth in Knowledge? CREOG Score Percent Correct Resident B PGY 1 2009 CREOG 195 PGY 2 2010 CREOG 195 PGY 3 2011 CREOG 193 55 4% 59 6% 65 ã To maintain the same 3 digit score each successive year – resident needs to answer 4‐6% more questions right than previous year (i.e. 12‐19 new facts). Assuming 320 CREOG questions and 1% is about 3‐4 questions. How Much Work is Involved in Improving CREOG Scores? ã Each year the average resident increases knowledge by 12‐19 new facts. ã To improve a comparative 3 digit score, the resident must do better than previously better scoring residents from previous years. ã The number of residents that they must outperform is dependent where their CREOG score is. 7 1/26/2014 How Many Residents Need to be Outperformed to Improve CREOG Scores 374 154 140 160 180 200 220 240 260 To go from 180 to 200, Resident A has to do better than 374 residents the next time. To go from 160 to 180, Resident C has to do better than 154 residents the next time. Additional Knowledge Needed to Improve CREOG Scores 20 Point Jump 140 160 180 200 220 240 260 Requires 32‐50 more correct answers than the previous year. Who to Remediate? Remediation Line 0.025 0.025 0.02 0.02 0.015 0.015 0.01 0.01 0.005 0.005 0 0 140 160 180 200 220 240 National CREOG Distribution 15% Line 260 ã Where do you draw the line at which everyone below is placed on remediation? ã Draw it high, then you work harder to remediate more residents ã Draw it low then you may jeopardize the weak resident passing ABOG ã ã ã ã 180 ‐ 16% 182 – 19% 185 – 24% 190 – 33% Scores are not linear 8 1/26/2014 Who to Remediate? ã SEM – This is an index of the examination’s precision in measuring your overall ability as represented by the score reported. If you were to repeat the CREOG In‐Training Examination 100 times, your score would fall in the range of +/‐ 1 SEM on 68 out of 100 times. Who to Remediate? Remediation Line 0.025 0.025 0.02 0.02 0.015 0.015 0.01 0.01 0.005 0.005 0 0 140 160 180 200 220 240 National CREOG Distribution 15% Line 260 ã ã ã ã 180 ‐ 16% 182 – 19% 185 – 24% 190 – 33% Consider everyone below this 9 Scores & Percentiles 2Score 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 Percentile 0.1% 0.2% 0.3% 0.4% 0.5% 0.7% 0.8% 0.9% 1.0% 1.1% 1.2% 1.3% 1.4% 1.5% 1.6% 1.8% 1.9% 2.0% 2.1% 2.2% 2.3% 3.0% 3.7% 4.3% 5.0% 5.7% 6.4% 7.1% 7.7% 8.4% 9.1% 9.8% 10.5% 11.1% 11.8 % 12.5% 13.2% 13.9% 14.5% 15.2% 2Score 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 Percentile 15.9% 17.6% 19.3% 21.0% 22.7% 24.4% 26.1% 27.8% 29.5% 31.2% 32.9% 34.6% 36.3% 38% 39.7% 41.4% 43.1% 44.8% 46.5% 48.2% 50% 51.7% 53.4% 55.1% 56.8% 58.5% 60.2% 61.9% 63.6% 65.3% 67% 68.7% 70.4% 72.1% 73.8% 75.5% 77.2% 78.9% 80.6% 82.3% 2Score 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 Percentile 84.1% 84.8% 85.5% 86.1% 86.8% 87.5% 88.1% 88.9% 89.5% 90.2% 91% 91.6% 92.3% 92.9% 93.6% 94.3% 95% 95.7% 96.3% 97% 97.7% 97.8% 97.9% 98% 98.1% 98.2% 98.4% 98.5% 98.6% 98.7% 98.8% 98.9% 99% 99.1% 99.2% 99.3% 99.5% 99.6% 99.7% 99.8% 99.9% CREOG Comparative 3 Digit Score Graph 0.025 0.025 0.02 0.02 0.015 0.015 0.01 0.01 0.005 0.005 0 0 140 160 180 200 220 240 260 National CREOG Distribution 15% Line National Mean
