AAMC GREAT Group Surveys, modified with permission
2009 Award for Excellence in Assessment Winner
Objectives: To create a valid assessment tool to evaluate the readiness of pharmacy students for advanced pharmacy practice experiences (APPEs). Methods: The Triple Jump Examination (TJE), created by the faculty of the College of Pharmacy, Touro University – California, is tailored to the four-year, two-plus-two curriculum of the college. The TJE consists of three parts: 1) a written, case-based closed-book exam, followed by 2) a written, case-based open-book exam, and 3) an Objective Structured Clinical Examination (OSCE). The TJE is administered at the end of each four academic semesters. Progression of students to APPEs is dependent on achieving a preset minimum cumulative (weighted average) score in the four consecutive TJE exams. Results: The predictive utility of the examination was demonstrated by a very strong correlation between the cumulative TJE scores and the preceptor grades in the first year (P3) of APPE rotations (r = 0.60, p>0.001). Reliability of the TJE was shown by strong correlations among the four successive TJE exams. A survey probing the usefulness of TJE indicated acceptance by both students and faculty. Conclusion: The TJE program is an effective tool for the assessment of pharmacy students’ readiness for the experiential years. In addition, the TJE provides guidance for students to achieve preparedness for APPE.
Beyond Formulas: Enhancing the Selection Process to Improve Assessment of Candidates for Admission to Pharmacy School.
Background: While traditional admissions criteria such as grade point averages, standardized test scores, and recommendation letters are important to consider, these factors provide only a limited picture of an applicant’s potential for success in a Doctor of Pharmacy program. Personal characteristics, such as attitudes and values shaped by one’s experience contribute as well to success or failure in such professional programs. However, admissions decisions can be challenging when the process goes beyond strict formulas. Unacceptable attrition rates in the Auburn University Harrison School of Pharmacy (AUHSOP) entering class of 2002 drove the School to re-evaluate the efficacy of its admissions process. Objective: Improve the admissions process to facilitate a more comprehensive approach to assessment of applicants. Method: The AUHSOP developed a list of expected attributes of entering students based upon the School’s curricular outcomes and educational philosophy. These attributes reflect ways of being that are not easily measured using traditional admission criteria. Included are: Humanistic Caring Skills, Professional Commitment, Confidence, Independence/Motivation, Adaptive Attitude, and Communication Skills. In collaboration with internal and external stakeholders, the AUHSOP developed an admissions process that utilizes a combination of informal interaction with multiple members of the School, formal individual interviews, and assessment of group interactions. Candidates for admission have the opportunity to interact with a broad range of people in the AUHSOP and are critiqued based upon personable attributes that would not be evident on paper. To enhance the efficacy of the admissions process, a rigorous self-assessment of individual interview questions, group activities, and evaluator training is conducted systematically. The Admissions Committee elicits feedback from faculty, staff, students, and alumni involved in the interview process. Benchmarks such as on-time graduation rates help to determine the strengths and limitations of the selection process. Results: Attrition rates have fallen markedly, on-time graduation rates have nearly doubled, and the rate of students completing the first year successfully has increased dramatically since restructuring of the admissions process.
An Annual Skill Based Mastery Exam as the Backbone for an Academic Assessment Program. Background: ACPE Standards 2007 challenges institutions to graduate students who possess the knowledge, skills and abilities required of competent entry level pharmacy practitioners. Few institutions have implemented a program to demonstrate skill mastery. Without an effective technique for demonstrating skill mastery, schools may not comply with ACPE Standards. Objective: Create a reliable, valid, and useful skill mastery assessment to inform curricular decisions and demonstrate compliance with ACPE standards. Methods: The faculty created Terminal Ability Based Outcomes (TABO) Statements. We selected a sample of these abilities to be tested by multiple choice exam. Minimal competency cut scores were calculated using a modified Angoff process. The assessment included 61 items for P1 students, 94 items for P2 students, 128 for P3 students, and 170 items for P4 students. The mastery score for the exam was calculated using the weighted average of the item cut-scores. A sub-score for each TABO was generated. A individual score report allowed each student to identify their unique areas of strength/weakness and remediate any deficiencies. Summary global reports provided a thorough analysis of curricular effectiveness. Outcomes: For the spring 2008 exam, 235 out of 239 examinees demonstrated overall mastery. All P3 and P4 students demonstrated mastery. The internal reliability of all four 2008 exams – P1 (0.65), P2 (0.81), P3 (0.82), P4 (0.80) –were satisfactory. Validity was enhanced by proper test design. Conclusion: A pharmacy school can develop a reliable, valid, useful, skill mastery exam tailored to the specific needs of the institution.
The College and the Dean are commended on their partnerships with stakeholders in the community, focused on pharmacist involvement in public health measures.
Six different team-based interdisciplinary case conferences (ICC) have been organized and were conducted in an interactive manner for the Pacific University College of Health Professions (CHP) students. Six schools from the CHP: pharmacy, dental health science, occupational therapy, physical therapy, physician assistant, and professional psychology participated in the case conferences. Students from all six schools attended the case conferences in which each conference lasted for 2 hours. A student survey was used to assess and improve the effectiveness of the case conferences and to identify the need for future interdisciplinary case topics.
Where possible, professionals from the community or patients themselves were included in presenting the case to further authenticate the information. Student round table discussions were broken into different segments throughout each case in which faculty representatives from the different disciplines facilitated the dynamic of group discussions. Active learning techniques were employed regularly.
Our results from student surveys indicate that students enjoyed the ICCs immensely and found the interdisciplinary cases and interaction with other health profession programs highly interactive and educative. While 85% of students found that the interdisciplinary cases will assist them in their future healthcare fields, 82% of the students, based on interdisciplinary communications, learned how other students from other health disciplines approached and solved health related cases.
In addition, a variety of ICCs’ strengths was identified by the respondents that can be used by colleges of health professions to organize ICCs:
“Learning Bridge” (LB) assignments which integrated didactic material with introductory pharmacy practice experiences (IPPE) were created for use by first-year PharmD (P1) students while at their IPPE sites. The LB assignments prepared by didactic faculty were distributed to both P1 students and preceptors prior to biweekly IPPE days. Case studies and other questions based on concurrent didactic material were formulated so as to require students to use pharmacy databases, drug information resources, and package inserts available at their experiential sites. Students were encouraged to discuss their thoughts with their preceptor to come to a therapeutically informed answer. Students and preceptors were surveyed with separate instruments following a series of LB assignments.
Student and preceptor participation in the Learning Bridge assignments was consistent and of high quality. The quantitative and qualitative survey results provided compelling evidence that both students and preceptors were enthusiastic about the LB assignments and agreed that the tool promoted student/preceptor interaction, self-directed learning, and critical-thinking skills. Students reported that the LB assignments increased their learning, knowledge of drug information, and ability to utilize package inserts. Both students and preceptors agreed that the LB assignments made IPPE training more productive.
Purposeful integration of didactic material with pharmacy practice is welcomed by both students and preceptors. Such assignments are not difficult to construct, give structure to student learning and student/preceptor interaction at the IPPE site, and increase student and faculty awareness of how the scientific content of the P1 curriculum is relevant to pharmacy practice. Our results indicate that the LB process integrates didactic and experiential realms, and the results were sufficiently encouraging to incorporate the LB process into our PharmD curriculum.
The Texas A&M Rangel College of Pharmacy (RCOP) experiential program consists of a variety of introductory and advanced experiences designed to provide the student with professional experience through the use of a structured and supervised program of study. Guidelines and policies for experiential practice are listed in appropriate program manuals and must be adhered to for the successful completion of the program. The RCOP offers introductory and advanced experiential courses under the philosophy of pharmaceutical and patient care to prepare students to function as independent, lifelong learners; to support their growth; and to strengthen their self confidence. The advanced practice experiences are designed to enable students to integrate and hone the skills learned in the first three years of pharmacy coursework. The ultimate goal of the experiential program is to produce a well-rounded, competent, caring, responsible, professional who can deliver exemplary pharmaceutical care and communicate effectively with diverse patients and colleagues.
Prior experiences in opening a new pharmacy academic program on a campus where professional programming was new led to the realization that healthy campus relationships, although extremely important, don’t just happen but must be deliberately and carefully developed. This report describes a dedicated attempt to integrate a pharmacy program into all aspects of the life of a traditional, undergraduate-only institution.
In February 2008, following two years of dedicated study, the Board of Trustees of Presbyterian College, a private, church-related, liberal arts college, published their resolution to establish the Presbyterian College School of Pharmacy (PCSP). Thus, the College officially began the process of starting the first graduate program in its history. The Board’s resolution directed the College’s President to search for a Dean and to appoint a transition team to work with him or her in the establishment of the new school.
In August 2008, the President appointed a Transition Team composed of the President, the inaugural Dean, and a select group of college employees chosen because of their interest in establishing the PCSP and their particular expertise. This group consisted of officers and directors of the College and a senior member of the undergraduate faculty; upon their hiring, two members of the Dean’s administrative team were added. In this manner and upon taking office, the new Dean of the PCSP was immediately presented with a temporary administrative team to assist him. Individual team member responsibilities included:
· development and ongoing refinement of a pro forma budget for the new program
· provision of a physical facility for the program, development of construction; plans, and the securing of a building contractor;
· submission of a detailed application for substantive change (the offering of a Pharm.D. degree) to the College’s regional accrediting agency;
· modification of the bylaws of the college to reflect the presence and influence of a new educational unit;
· development of local and regional support and capital funding for the PCSP;
· recruitment of PCSP faculty and staff; and
· publication of newsworthy developments in the formative progress of the PCSP.
Weekly team meetings consisted of reports from team members concerning the progress of their assignments; lively discussions yielding suggestions and follow-up actions ensued.
After six months of planning, a larger Pharmacy Integration Group was formed to address transition issues for the PCSP. This larger group was composed of both PC and PCSP personnel and was tasked with resolving and implementing pharmacy-related processes into a common campus structure. Smaller task forces met regularly to address such matters as: academic services (registrar, student records, pre-pharmacy advising, library), student services (counseling, honor council, health), IT, admissions/financial services, housing, communications/public relations, events, food services, security, athletics, and facilities.
The consensus is that the described process assisted in and expedited the successful establishment of the PCSP and its integration into the Presbyterian College community. The teams functioned efficiently for two years and progressively yielded their function to the administration and faculty of the Presbyterian College School of Pharmacy.
The School of Pharmacy at the Lebanese American University (LAU) mapped its newly implemented curriculum to the curricular content suggestions in Appendix B of the Accreditation Council for Pharmacy Education (ACPE) Standards, and to the Educational Outcomes of the Center for Advancement of Pharmaceutical Education (CAPE).
A dynamic approach of curricular evaluation was undertaken in a form of continuous quality improvement that is driven by the School’s mission. In order to ensure a common vision of the curriculum and achieve a comprehensive mapping of curricular content, all School faculty members were engaged in the mapping process through tools preparation and/or mapping the course they teach and/or coordinate. For every course, a subcommittee that included the course coordinator, course faculty, and a Curriculum Committee member was formed in order to map the corresponding course content and submit its findings to the Curriculum Committee. Regular meetings were held between Curriculum Committee members and various subcommittees. During these meetings, respective CAPE Supplemental Educational Outcomes and curricular content in Appendix B were shared and concepts of ability based outcomes, domains, competencies, dimensions, and metrics were discussed.
Courses that were related or complementing each other in content were mapped collectively after being grouped into six streams: Pharmacy Practice; Social, Behavioral, and Administrative Sciences; Basic Biomedical Sciences; Pharmaceutical Sciences; Pharmacotherapeutics; and Library Educational Resources. Maps were developed to include courses within a stream along with their respective CAPE Supplemental Educational Outcomes and the curricular content in Appendix B. A total of six maps were generated. When detailed CAPE educational outcomes were not available for a course or a sequence of courses, a taskforce was assigned to develop such tools. Metrics adopted in mapping the competencies were labeled as “Meet”, “Do Not Meet”, or “Partially Meet”. Mapped courses were categorized for their competencies under the three levels of: “Introduce, Reinforce, and Apply”.
In order to ensure a comprehensive curricular review and that competencies are met at multiple levels of “Introduce, Reinforce, and Apply”, courses within a series were first mapped vertically (one course mapped across multiple competencies) then horizontally (one competency mapped across multiple courses). After each stream was mapped, a report was generated and included recommendations and follow up tables detailing the required actions for recommendations of mapping implementation and responsible party. The mapping process clearly indicated at which level(s) of the curriculum a competency was delivered and also identified curricular gaps and content overlap. The final mapping report was reviewed at a faculty retreat whereby recommendations were generated and a plan of action for implementation was developed.
The plan is to repeat the mapping cycle for every stream once every 3 years, unless changes in the ACPE Standards or CAPE educational outcomes, or curriculum assessment outcomes warrant earlier evaluation.
The primary goal of the Interprofessional Education (IPE) program at the Lebanese American University (LAU) is to bring together LAU healthcare professions students to learn how to collaborate across disciplines, and ultimately enhance the quality of patient and client health and social care. Therefore, a workgroup of 13 faculty members from the Schools of Nursing, Pharmacy, Medicine, and Arts and Sciences (nutrition and social work) was formed with its mission to establish an IPE program that uses a student-centered educational approach, building on the curricula of LAU’s health and social care programs.
The IPE workgroup meets regularly to discuss, develop, and plan IPE activities. A course list was created on Blackboard™ to facilitate literature sharing. Common educational themes included: Communication; Compliance and Adherence; Conflict negotiation; Ethics in clinical practice; Evidence-based medicine; Health promotion; Patient safety; and Sexuality. These topics will be presented to students in mini lectures, simulations, and case discussions. All students will have multiple IPE learning experiences during their enrollment. The IPE workgroup also held a retreat to share the IPE goals, mission, and plans with the rest of faculty. In addition, faculty and students of participating schools were surveyed for their readiness and interest in the IPE program. Survey results showed high interest and support for the IPE program, its goals and objectives. Furthermore, LAU has included the IPE program initiative in its 2011-2016 Strategic Plan, which substantiates the value and commitment of LAU leaders to IPE.
To streamline course delivery and considering students’ level of readiness (e.g., clinical experience; familiarity with interdisciplinary practice), students will be grouped into three “steps”: 1, 2, and 3. In Step 1, students who have not yet had clinical learning experiences will be brought together for learning activities, regardless of their class year. Step 2 students are those enrolled in their initial clinical experiences. Step 3 students are those nearing graduation with extensive clinical experiences and broader knowledge of healthcare. IPE is not limited to classroom and laboratory settings. Pharmacy, medicine, and nursing students and faculty are collaborating in the clinical setting using an interprofessional collaborative practice model at a Volunteer Outreach Clinic that serves a select population in an underserved area. This experience is helping students learn about each other’s roles and how to work together in identifying and addressing patient care needs.
Assessment and evaluation are important aspects of the IPE program. Students and faculty will be asked to complete evaluations for their IPE learning activities. Instruments that have been utilized by other IPE programs will be used to assess the overall impact of the IPE program on students learning. Data generated from these assessments will guide the future direction of the IPE program.
One of the most rewarding aspects of our experience is the enthusiasm that is generated among the faculty and students across our health and social care programs. Faculty and students are learning how to work together more effectively, all for the benefit of health and social care patients and clients.
Students are assigned to complete medication reconciliation P3 for all patients within three days of admission to the medicine faculty hospitalists (MFH) service. Each student is assigned to cover a specific patient care area and must complete medication reconciliation for all MFH patients meeting criteria in his or her assigned area; the six students are expected to work as a team to divide the workload fairly and ensure that all patients are covered. Each day, the students must maintain a list of patients for whom medication reconciliation was completed and communicate that information to the students scheduled for the next day (students working on Monday begin by reviewing data on patients admitted on the previous Friday).
The Direct Patient Care IPPE is a valuable learning activity that not only meets the requirements of ACPE but also meets the needs of the college of pharmacy in providing an experience that gives students real patient care responsibility, thus providing better preparation for inpatient clinical rotations during their fourth-year advanced practice experiences. Furthermore, the students perform a needed service for the health system and patients, accomplishing meaningful work that would otherwise have to be done by others. The IPPE is truly a win/win/win for students, the College, and the Hospital.
Paul C. Walker, Kathy S. Kinsey, Michael D. Kraft, Nancy A. Mason, John S. Clark. Improving student education and patient care through an innovative introductory pharmacy practice experience.Am J Health Syst Pharm April 15, 2011 68:655-660; doi:10.2146/ajhp100490
The comprehensive assessment plan utilized by the Auburn University Harrison School of Pharmacy (AUHSOP) is a continuous quality improvement (CQI) process driven by its Strategic Planning Committee and primarily supported by its Office of Teaching, Learning and Assessment (OTLA). On an annual basis, the Strategic Planning Committee assesses the School’s progress on its Strategic Initiatives and revises its plan, developing new initiatives and prioritizing them accordingly. This process includes input from faculty and staff, usually gathered at the annual faculty retreat. Student input is included as well through survey data and through student representatives who attend the faculty retreat. The OTLA provides necessary data to be examined in the assessment process. The process is also supported by the Professional Education Committee (PEC), which provides assessment data regarding the curriculum through such activities as course reviews and curricular mapping, and the Accreditation Self Study Committee, which is responsible for guiding the self-study process and writing the self-study report at the appropriate time in the School’s accreditation cycle. The CQI process continues on an annual basis in the AUHSOP regardless of the timing of accreditation site visits and reporting deadlines.
At the time the School established a new curriculum (implemented in 2005), the AUHSOP developed internal criteria for each of the 30 ACPE Standards and began developing tables of measures for each of the criteria by identifying instruments, timelines, benchmarks and goals. The criteria reflect aspirations that surpass the expectations required to meet accreditation standards and could even be characterized as somewhat idealistic. We recognize that we may always fall somewhat short of our own criteria, but those criteria are in place to push us towards a higher standard. In addition to the ACPE Standards, we have developed two internal standards, one for graduate programs and one for scholarship, which allow us to assess additional areas of our mission.
Part of the assessment plan includes assessing the plan itself. In addition to assessing our performance on the ACPE standards and our additional internal standards, faculty and staff are asked for feedback on the process. For example, they are asked to note any criteria that may need revision and to comment on how the existing criteria may be revised for greater clarity and appropriateness. They are also asked for suggestions on how to improve the processes and mechanisms utilized to gather input. Such information is fed back to the Strategic Planning Committee through the OTLA and utilized to revise the assessment plan as needed.
The Post Course Review (PCR) is a course “look back” process that is required for all School of Pharmacy courses within six weeks of the completion of the semester in which the course is taught. Using a standardized form, it is completed by the course team and submitted to the Curricular Affairs Committee for review, and is subsequently available to the following year’s course team during the course planning stage.
The purpose of the PCR is to provide timely reflection and feedback on course delivery that serves as a basis for course refinement by future teams. In doing this it supports continuity within the course despite possible changes in team leader or team members and provides a documented record of any minor changes that occur within the curriculum from year to year.
The team describes efforts in the areas of content, delivery methods, evaluation, etc… including strategies to promote active learning, and also reports any minor changes implemented and the impact of those changes. The team also provides a summary of, and reaction to, student course evaluations. In addition, the team reports the overall outcomes of the course, including key statistics such as number of learners, course mean and standard deviation, number of failures, and number of second chance exams taken/passed. Lastly, the team provides an accounting of the number of lectures or course activities (total and by team member), and estimates the percent contribution by each team member. Based on this information, the team has the opportunity to recommend any modifications for the improvement of the course for the next year, as well as suggest modifications to other courses in the curriculum that directly relate to the delivery of the course reviewed (examples of the latter might include prerequisite requirements or changes in content to avoid gaps or duplication of material).
Texas Tech University Health Sciences Center, School of Pharmacy. 2011