|
|
||||||||
1 From the Department of General Practice (HJSM and JWVR), the Skillslab Department (J-JER), and the Department of Educational Development and Research (LWTS), Maastricht University, Maastricht, the Netherlands; and the Department of Gastroenterology, University of Amsterdam (LMHM-V).
2 Presented at the Third Heelsum International Workshop, held in Heelsum, the Netherlands, December 1012, 2001.
3 Supported by research grants from the Dutch Sugar Bureau, the Dutch Dairy Foundation for Nutrition and Health, and the Dutch Meat Board.
4 Address reprint requests to HJS Maiburg, Maastricht University, Department of General Practice, Paviljoen DEB 1, PO Box 616, 6200 MD Maastricht, the Netherlands. E-mail: bas.maiburg{at}hag.unimaas.nl.
| ABSTRACT |
|---|
|
|
|---|
Objective: The study objective was to assess the degree to which computer-based instruction on nutrition improves factual knowledge and practice behavior of general practitioner (GP) trainees.
Design: We carried out a controlled experimental study, using a 79-item knowledge test and 3 incognito standardized patients visits in a pre- and posttest design with 49 first-year GP trainees. The experimental group (n = 25) received an average of 6 h of a newly developed computer-based instruction on nutrition. The control subjects (n = 24) took the standard vocational training program.
Results: The percentage of correct answers on the knowledge test increased from 30% at pretest to 42% at posttest in the experimental group, and from 36% to 37% in the control group. Analysis of covariance, with the pretest scores as covariate, showed a significant experimental versus control group difference at posttest: 9.2% (P = 0.002). The mean percentage of correctly performed items during the 3 standardized patients visits (assessed by checklists) showed an increase in the experimental group from 20% at pretest to 36% at posttest, whereas the control group changed from 20% to 22%. Analysis of covariance, with the pretest scores as covariate, revealed a significant group difference at posttest: 13.7% (P < 0.001).
Conclusion: The computer-based instruction proved its effectiveness, both by increasing factual knowledge and by substantially enhancing GP trainees practice behavior on the subject of nutrition.
Key Words: General practice GP trainee computer-based instruction nutrition guidance standardized patients knowledge test
| INTRODUCTION |
|---|
|
|
|---|
Although the value of CBI has been generally established and accepted (15, 16), we still had to prove the effectiveness of the newly developed CBI on nutrition. This assessment formed part of the NECTAR (Nutrition Education by Computerized Training and Research) study, whose main objective is to enhance the knowledge, skills, and attitudes of Dutch GP trainees on the subject of nutrition.
In terms of the effect of the nutrition CBI as an educational tool, we were interested in the improvement of both knowledge (competence) and performance of GP trainees (17, 18). Competence-based assessment measures what doctors can do in controlled representations of professional practice, whereas performance-based assessment measures what doctors actually do in their professional practice. Our first objective was to determine the degree to which GP trainees factual knowledge of nutrition would improve as a result of the CBI. Second, we wanted to examine whether the nutrition CBI improved the adequacy of consultations performed by GP trainees in real practice with patients with nutrition-related complaints.
The first objective refers to the "knows" level of facts and figures, as distinguished by Miller (19, 20). Many assessments of the value of educational tools are restricted to this level. The second question refers to the "does" or "performance" level, which implies transfer of knowledge and skills into daily practice.
| SUBJECTS AND METHODS |
|---|
|
|
|---|
Subjects
We included all 49 GP trainees who started their GP vocational training program in 2000 at Maastricht University. The selection committee of the Department of Vocational Training, in which the researchers were not represented, allocated these trainees to 4 groups. Two groups were to constitute the experimental group (n = 25), with the other 2 groups being control subjects (n = 24). Together with the selection committee, we stratified the admitted GP trainees according to their sex and the university where they had completed their undergraduate medical program (Maastricht vs elsewhere). Afterward, the selection committee reallocated some GP trainees, mainly for logistic reasons beyond the scope of our study.
At the start of their vocational training, GP trainees received information about the design of the study and the way they would be informed about its progress and results. During the same session, they completed a questionnaire on background variables (mainly focusing on previous experience with the subject of nutrition), providing us with information about the composition of the research groups. Participation in the experiment was obligatory.
All GP trainees worked in different practices with their own GP trainer (one trainee per trainer). Therefore, we also asked their GP trainers to fill out a questionnaire assessing their own attitude about nutrition, as GPs and as GP trainers. We argued that differences in the way GP trainers handle nutrition-related problems in practice, as well as differences in the numbers of patients with these problems seen by the GP trainers, might well influence GP trainees and thus to a degree confound the effect of the CBI.
Intervention
The GP trainees in the experimental group received a newly developed CBI, involving a simulation of a GP consultation consisting of 12 interactive streaming-video patient cases presenting with complaints related to nutrition (ie, food pyramid, obesity, diabetes mellitus, hypercholesterolemia, hypertension, irritable bowel syndrome, and constipation). The teaching objectives of the CBI are shown in Table 1
. The GP trainees ran the program during their weekly vocational training days at the institute. We scheduled the CBI sessions at 2- to 3-wk intervals, with a maximum of 5 sessions, each session taking 1.5 h. After a plenary instruction session by the researcher, the GP trainees went through the CBI at their own pace during these sessions, each GP trainee being provided with a personal computer. The actual learning process with the CBI took an average of 6 h. The control group took the regular vocational training program, in which nutrition topics are sparsely incorporated and are subordinated to medical diagnosis and treatment.
|
|
We performed a pilot of the knowledge test in a GP trainee group not included in our study to assess the level of difficulty of the test and to avoid ceiling effects, as we planned to use the same test twice. The GP trainees took the test under exam conditions. The researcher graded all tests following a predefined grading protocol, being blinded for group allocation. The test score consisted of the number of correctly answered items (incorrect answers and items not answered scoring zero), converted to percentages.
Instruments: incognito standardized patients
To assess the practice behavior of GP trainees with regard to problems on the subject of nutrition, we decided to use incognito standardized patients (SPs). SPs are nonphysicians taught to portray patients in a standardized and consistent way. Such incognito SPs visit GP trainees as regular patients during daily practice and complete a checklist on the consultation afterward, thus assessing the "performance" level of the trainee (25, 26). We trained 14 newly acquired SPs, who matched the intended roles. Their instruction included standardized role-playing, reliable completion of checklists, and remedial training halfway through the planned visits. The total training time amounted to 16 h per role.
The content of the 3 SP roles in the pre- and posttest (intestinal problems, cardiovascular disease/hypertension, and overweight/ diabetes mellitus) covered the entire domain of nutrition as incorporated in the CBI. We developed the SP checklists with the experts who had contributed to the content of the CBI. The total number of checklist items in the pretest was 118; in the posttest it was 115. Scoring categories for SPs included "performed correctly" (= yes), "not performed/performed incorrectly" (= no), and "dubiously performed," as shown in Figure 2
. SPs completed the checklists immediately after leaving the GP practice. In the case of a follow-up consultation, SPs used the same copy of the checklist again. The score for each case consisted of the number of correctly performed items (incorrectly, dubiously, or not performed items scoring zero), converted to percentage scores. We computed the total SP test score as the mean of the 3 checklist scores.
|
Statistics
To evaluate differences in baseline variables between research groups, we used an independent samples t test for age, Mann-Whitney tests for (skew-distributed) numeric variables and ordinal variables from the GP trainers questionnaires, and chi-square tests for data on sex and the university of undergraduate medical studies. To assess whether differences in knowledge test and SP test scores between experimental and control groups in the posttest were significant, taking into account the differences in the pretest, we carried out a linear regression analysis (analysis of covariance) (29). We also used the linear regression analysis to elicit and correct for confounding variables. To compute the significance of differences in knowledge test and SP test scores between the pre- and posttests, we performed paired samples t tests for both experimental and control groups.
Depending on the variable level, we used Pearson r or Spearmans
for correlations. We computed the reliability of the instruments as Cronbachs
. For the knowledge test, we entered the scores on all test items to compute Cronbachs
; for the SP test, we entered the percentage scores of the 3 cases.
Two-sided P values
0.05 were considered significant. We carried out all analyses in SPSS for Windows 9.0 (SPSS Inc., Chicago, IL) (30).
| RESULTS |
|---|
|
|
|---|
|
Knowledge test
The results of the knowledge test, including total scores and scores per chapter are shown in Table 3
.
|
Incognito SPs
Of the 294 SP visits planned, 287 were actually carried out (98%). The 7 missing visits were due to dropout by GP trainees, as mentioned above. Five of the 287 visits resulted in a follow-up consultation within several days, mainly to discuss a requested food record.
The results of the SPs pre- and posttest scores are displayed in Table 4
. Based on all visits, the SP scores for the experimental group between the pre- and posttest showed a relative improvement of 79.6%. For the sake of consistency in our method of analysis, we once again used analysis of covariance to compute experimental versus control group differences on the posttest, although the pretest scores hardly differed. This analysis showed a significant difference of 13.7% (P < 0.001) between the groups at posttest. Relative to the mean pretest score, this implied an improvement of 68.2% in the experimental group. Reliability of the SP pretest was 0.67, and that of the posttest 0.72 (experimental group 0.45; control group 0.71).
|
Confounding variables
Analyzing for confounding variables, we found 2 items on the GP trainers questionnaires on which the trainers of GP trainees in the experimental group had scored significantly higher than the trainers of the control group GP trainees (Table 2
). The first item, "perceived importance of nutrition in GP training," did not significantly correlate with posttest scores on the knowledge test (r = -0.186; P = 0.204) or the SP posttest score (r = 0.244; P = 0.099). "Use of obesity brochure" showed significant correlations with neither the knowledge test (r = 0.180; P = 0.221) nor the SP test (r = 0.223; P = 0.115).
We also found a significant difference of 5.8% (P = 0.001) in SP pretest scores between male and female GP trainees and a significant difference of 4.4% (P = 0.024) between GP trainees who had graduated from Maastricht University and those who had graduated from other institutions. Introducing sex and the university of undergraduate medical studies into the analysis of covariance showed that these variables were not confounding, as they hardly caused any change in the experimental versus control group difference on the SP posttest (31).
| DISCUSSION |
|---|
|
|
|---|
The results of the knowledge test are encouraging, although it must be acknowledged that the questions in this test concentrated on straightforward facts. Facts and figures about nutrition are provided in the CBI in "pop-up" screens, which can be activated on demand during the session, and in concluding tutorials. It is debatable how eagerly GP trainees searched for these elements in the CBI. Some GP trainees in the experimental group stated their disappointment on being questioned on facts and figures in the posttest, as they had not paid much attention to these elements in the CBI in their search for practical tools. A semiannual Dutch national knowledge test for GP trainees provides some comparative material on the development of GP trainees knowledge about nutrition, as induced by the CBI (24). This national knowledge test, covering the total GP domain, shows an average growth in knowledge among all first-year GP trainees that is quite similar to the growth in knowledge about nutrition we found in our experimental group GP trainees. This suggests that knowledge about nutrition in our control group GP trainees fell short of their knowledge about the total GP domain.
The results of the performance test taken by our GP trainees are remarkable for more than one reason. First, we succeeded in establishing an improvement in consultation behavior, using incognito SPs as an instrument. Although several studies have been conducted using SPs in practice, effects on practice behavior were rather ambiguous (32, 33). Second, the practice behavior itself was found to improve, indicating that the nutrition CBI is suitable for achieving this relatively ambitious target. We attribute this outcome to the concept of the CBI being a simulation of GP consultations. As GP trainees seem to be focused on the acquisition of tools for daily practice, the program provides them with not only written guidelines for nutrition history taking and nutrition guidance but also video clips of GP experts as role models for good nutrition practice.
The SPs pretest scores, as a measure of current practice behavior on the subject of nutrition, confirmed our expectations. These expectations were based on previous research among GPs and GP trainees, indicating a lack of nutrition education and a lack of sufficient consultation time as important barriers to performing nutrition guidance (12). The CBI seemed to remove both barriers. Clearly, it offers sound nutrition education. Furthermore, our results suggest that GP trainees overcome the barrier of the lack of sufficient consultation time as soon as they have acquired the necessary practice skill of nutrition history taking and nutrition guidance.
The question may be raised whether the absolute level of practice behavior achieved after our intervention is relevant for daily practice. Some idea of this can be obtained by comparing the scores of our GP trainees, whose mean score was 36%, with those of experienced GPs found in previous studies with SPs, showing mean scores of 55% (17, 34). Of course, we have to take into account that these studies belonged to a different domain. Nevertheless, our findings show that the CBI did help GP trainees to make considerable progress toward reaching experience levels found among practicing GPs.
Whereas GPs likely take several consultations to deal with patients complaints thoroughly (35), the vast majority of the SPs visits turned out to be one consultation only, a situation partly accounted for by the limited time available in the study. This means that the results of our study have to be categorized as short term and based on one consultation.
With regard to the composition of the experimental and control groups, we conclude that we have managed to achieve adequate similarity in terms of characteristics and SP pretest scores. However, we have to take into account the relatively small number of subjects included, causing rather wide confidence intervals.
We wondered whether our study assessed the real performance of our GP trainees, as the detection rate of SPs was somewhat higher than that found in other SP studies (36). However, the scores found in the analysis of only undetected visits hardly differed from those based on all visits. The effect on practice behavior, within the experimental group as well as between the experimental and control groups, was firmly sustained in the subgroup of undetected visits. As GP trainees were visited by SPs in their own consultation rooms, during regular consultation hours, we may conclude that our assessment of GP trainees in practice, based on all visits, very closely reflects daily practice performance.
Finally, to return to the educational starting point of the study, the merits of the nutrition CBI in the education of GP trainees on the subject of nutrition are obvious. What is noteworthy is not only the effects of the program, as discussed above, but also the fact that the GP trainees used this program without any teacher intervention. The CBI permits self-directed learning by GP trainees, a learning method that is one of the basic elements of a new curriculum to be implemented in Dutch GP vocational training. Furthermore, as models of nutrition guidance do not differ between GPs and GP trainees (12), it should also be possible to use the CBI for refresher courses for practicing GPs. In terms of intrinsic motivation, similar or even better effects might be anticipated, given that GPs may voluntarily choose to attend the program, whereas GP trainees had the program imposed on them.
For many years, teachers and trainers have entertained great hopes for using information and computer technology for educational purposes. We hope our study will contribute to speeding up the changes in the educational landscape.
| ACKNOWLEDGMENTS |
|---|
There are no conflicts of interest.
| REFERENCES |
|---|
|
|
|---|
This article has been cited by other articles:
![]() |
K. C Lindell, K. M Adams, M. Kohlmeier, and S. H Zeisel The evolution of Nutrition in Medicine, a computer-assisted nutrition curriculum. Am. J. Clinical Nutrition, April 1, 2006; 83(4): 956S - 962S. [Abstract] [Full Text] [PDF] |
||||
![]() |
J. J van Binsbergen and A. J. Drenthen Patient information letters on nutrition: development and implementation Am. J. Clinical Nutrition, April 1, 2003; 77(4): 1035S - 1038. [Abstract] [Full Text] [PDF] |
||||
![]() |
A S. Truswell, G. J Hiddink, and J. Blom Nutrition guidance by family doctors in a changing world: problems, opportunities, and future possibilities Am. J. Clinical Nutrition, April 1, 2003; 77(4): 1089S - 1092. [Abstract] [Full Text] [PDF] |
||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |