Excellence in Internal Quality Assurance: An Empirical Study in United States

Assessment or internal quality assurance (IQA) processes have often been driven by external stakeholders such as accreditation and governmental agencies of higher education, which are focused on accountability rather than quality improvement. This research examined how private and non-profit Doctoral and Research institutions with less public financial dependence and accountability requirements adopted the Excellence in Assessment (EIA) rubric to improve their IQA models that supports improvement. A survey based on National Institute of Learning Outcomes Assessment (NILOA) EIA rubric was sent to the ASSESS Listserv to learn if there was a difference in EIA scores related to source of funding and Carnegie characteristics. A two-way MANOVA analysis of the survey responses showed that there was no difference in IQA practice followed EIA rubric between source of funding (public and non-profit private institutions) and Carnegie classification (Research and Comprehensive). Recommendations are made regarding the reliability of the EIA rubric. The EIA designation can serve as a framework for U.S. and non-U.S. higher education institutions to benchmark and improve the current IQA processes.


INTRODUCTION
U.S. higher education institutions face increasingly numerous questions about their value and return on investment, as well as the effectiveness and efficiency of instructional practices (Emil & Cress, 2013). This public discourse has included diverse communities of interest from stakeholders such as students and their families and public sectors including donors, employers, and governmental agencies. As a result of the Spellings Commission (U.S. Department of Education, 2006) argument of growing evidence of shortfalls in the quality of student learning outcomes, quality agenda emerged to ensure academic quality is attended, measured and improved (Ewell, 2012). To address internal and external stakeholders' increasing concerns, U.S. higher education institutions have increased emphasis on reliable and effective quality assurance processes to advance student learning (Harmanani, 2016;Emil & Cress, 2013;Jankowski & Provezis, 2011;Ewell, 2009).
The Spellings Commission recommended that institutions provide transparent data and consumer information to "potential buyers" across institutional, national and international sectors to support students and families when selecting a prospective institutionsimilar to shopping for a commodity such as a car (U.S. Department of Education, 2006). Transparency and accountability typically refer to activities by institutions aimed at holding providers responsible for their performance through the disclosure of comparative completion and graduation rates at the institutional and program levels, educational processes, and outcomes to better inform prospective students and other stakeholders (McCormick, 2010). Policies related to transparency and accountability are widely adopted strategies to drive quality improvement and stimulate consumer choice (Kurtzman, 2010). Further transparency in assessment processes have been implemented to provide explicit information, such as learning outcomes, benchmark statements, and assessment criteria, which communicate institutional commitment to internal quality assurance (IQA). Given  55  these conditions, National Institute of Learning Outcomes Assessment (NILOA) utilizes the regional accreditor (Higher Learning Commission) definition of term "assessment" as: A participatory, interactive process that provides data institutions need on their students' learning, engages the college and others in analyzing and using that information to confirm and improve teaching and learning, produces evidence that students are fulfilling the outcomes the institution intended, guides colleges in making educational and institutional improvements, evaluates whether changes made improve/impact student learning, and documents the learning and institutional efforts (NILOA, 2019).
With respect to HEIs, this definition of "assessment" is closely aligned to the term "quality assurance" commonly used in Europe and Asia (Fuller & Skimore, 2014). In this research study, NILOA's term, "excellence in assessment" (EIA) is used interchangeably with internal quality assurance.
At the national level over the last two decades, a number of important transparency and accountability initiatives have been created, making impactful changes on IQA practices in U.S. higher education. First, national higher education association leaders and the U.S. federal government created initiatives to provide desired consumer information publicly on websites such as the College Navigator Web site, National Center for Education Statistics and Voluntary Accountability System (VAS) (Russell & Markle, 2017). The second initiative supporting accountability and transparency has been led by national and international professional organizations, promoting authentic assessment to assess students' knowledge, skills and attitudes. A commonly used authentic assessment measure is the Valid Assessment of Learning in Undergraduate Education (VALUE) proposed by the American Association of Colleges and Universities (AAC&U) with a set of 16 learning outcomes (AAC & U, 2015).
The third initiative is the Transparency Framework developed by the National Institute for Learning Outcomes Assessment (NILOA) in 2011. The Transparency Framework aims to help institutions with the entire IQA cyclical process, through articulating six core components relevant to IQA, including student learning outcomes statements, assessment plans, assessment resources, current assessment activities, evidence of student learning and use of learning evidence. To recognize the institutions' efforts to work on effective IQA practices, in 2016 NILOA named the first institutions with the Excellence in Assessment (EIA) designation; the EIA Designation is co-sponsored by the Voluntary System of Accountability (VSA), NILOA and AAC&U. Institutions submitted narratives of up to 3000 words with supporting evidence to demonstrate their IQA practices evaluation which was conducted via the EIA rubric utilizing eight themes and 33 items (NILOA, 2019). These three initiatives (EIA, VSA and AAC&U) align with Stosich, Bae & Snyder's (2018) argument that stakeholders emphasize accountability and transparency policies for systemic improvements using multiple measure approaches and performance assessments.
By the late 1980s, as U.S. regional accreditors embedded "institutional effectiveness" into the accreditation standards, institutional efforts transitioned from IQA for accountability to IQA for quality improvement (Gaston, 2014;Ewell, 2009). As a component of IQA for quality improvement, assessment involves establishing goals, selecting suitable measures, gathering data of goal attainment and using the data for institutional improvement (Banta et al., 2009;Suskie, 2009;Walvoord, 2004). This operational definition emphasizes assessment for improvement as opposed to solely accountability practices. Ewell (2009) clarified that the accountability paradigm centers around comparisons across institutions or programs, or against fixed standards of performance. The quality improvement paradigm includes tracking progress over time or against established internal (institutional) goals. Student learning outcomes assessment focuses on authentic assessment based on expected learning outcomes connected with actual student assignments and work. The objective is to move assessment practices toward an institutional culture that provides both external accountable information and internal quality improvement (Jankowski et al., 2018). However, in a survey with chief academic officers conducted by NILOA in 2017 by Jankowski et al. (2018), about 50% of the respondents reported that assessment data was primarily used to prepare for program and institutional accreditation rather than informing resource decisions, admissions, transfer policies, academic decisions by faculty, and academic staff performance appraisals (Harmanani, 2016).
From a public policy perspective, funding is used to "coerce" or "incentivize" specific outcomes. Thus, in the U.S., governmental funding has been utilized to incentivize regional and now national accreditation in order to ensure fundamental characteristics for publicly-funded higher education institutions (HEIs). These characteristics are related to faculty qualifications, student learning outcomes and graduation rates, learning support and resources, facility resources, financial solvency, shared governance between faculty and administration, and fiduciary responsibilities of the governing board. Therefore, previous research has argued that public HEIs developed their IQA practices to address stakeholder accountability requirements such as accreditation and governmental funding criteria rather than

VIETNAM JOURNAL OF EDUCATION
 56  quality improvement. Meanwhile, due to financial portfolios that depend less on federal and state funding, private and non-profit HEIs have more independence when building IQA models. Given this context of tension between externally driven accountability initiatives and internally directed IQA cultures, the researcher was interested in examining how private and non-profit Doctoral and Research institutions with less public financial dependence and accountability requirements adopted the EIA rubric to improve their IQA models that supports improvement.
The student learning outcomes assessment movement has been in place in the U.S. for the past forty years (Jankowski et al., 2018;Martin, 2017) and some empirical research exists on IQA culture in the U.S. (Ndoye & Parker, 2010;Fuller et al., 2015;Fulcher et al., 2016;Jankowski et al., 2018). However, with the 2016 introduction of the Excellence in Assessment (EIA) designation, the opportunity is presented to examine the relationship between public funding and Carnegie Classification with the EIA designation. Therefore, this study sought to address the following research question: Is there a difference in the EIA application description on institutional IQA activities based upon institutional control characteristics, specifically public, for-profit private and non-profit private and Carnegie classification (Research, Comprehensive and Baccalaureate)? The study had one hypothesis: There is a difference in EIA rubric responses in the following categories: diverse stakeholders' engagement, evidence of student learning, use of evidence of student learning and reflection and growth/improvement plans among public, for-profit private and nonprofit private institutions and Research/Doctoral and Master/Comprehensive and Baccalaureate.

LITERATURE REVIEW
The NILOA EIA rubric provides the conceptual framework for this research study. Of the rubric's eight dimensions, four specifically focus on the quality improvement and were examined in the survey, namely "groups and individuals engaged in assessment activities", "institutional-level evidence of student learning", "use of institutional-level evidence of student learning" and "reflection and growth/improvement plan". This literature review discussed the relevant researches about the four components of the EIA rubric.

Groups and individuals engaged in assessment activities
For diverse stakeholders' engagement in assessment, faculty engagement is essential to IQA success. This requirement aligns with the Council for Higher Education Accreditation (CHEA) emphasis on engaging faculty in all aspects of formative and summative assessment at both the organizational and classroom levels to provide evidence of student learning (Emil & Cress, 2013). Common challenges revealed by multiple IQA research studies have been bureaucracy in documentation and procedures (Cardoso et al., 2019), distraction of faculty from major responsibilities in teaching and research (Tavares et al., 2017) and invisible evidence of quality improvement (Vukasovic, 2014). As a result, institutions have experienced faculty resistance to IQA activities (Baas, Rhoads, & Thomas, 2016).
Focusing on engaging faculty in IQA activities, some research studies have identified good practices such as embedding the assessment process in the curriculum and co-curriculum expectations for quality (Ndoye & Parker, 2010;Robinson et al., 2019;Stitt-Bergh et al., 2019;Schoepp & Tezcan-Unal, 2016) and becoming a valued part of planning and teaching beyond getting a good report from the accreditation team (Meredith, 2013). Faculty leadership in the IQA process is another good practice because it spreads the level of IQA expertise and increases consistency among the colleges (Baham, 2019;Verzinski et al., 2019;Schoepp & Tezcan-Unal, 2016). In addition, institutions can use helpful outcomes assessment books from assessment experts (Allen, 2004;Astin & Antonio, 2012;Banta & Palomba, 2015;Banta & Black, 2009;Kuh et al., 2015;Suskie, 2009) to build effective professional development workshops to facilitate faculty comprehension of the IQA process (Price et al., 2011;Verzinski et al., 2019).
Student engagement also plays an important role in diverse stakeholders' engagement in IQA (Verzinski et al., 2019). A Provost survey at 811 regionally accredited, U.S. undergraduate degree granting institutions conducted by Jankowski et al. (2018) found that 76% of the institutions used the national student survey in the IQA process to improve instruction and services. Another common practice is to engage students in course evaluations to provide faculty feedback for instruction improvement (Linse, 2017;Stanny & Arruda, 2017). Cook-Sather, Bovill & Felten (2014) suggested that student partnership is the highest level of student engagement in the IQA process to enhance engagement, improve metacognition and enhance learning experiences. They asserted that student partnership requires collaboration, provides an opportunity to engage and acknowledges that different players bring different contributions to the table. Still, Curtis (2018) argued that NILOA EIA designation does not emphasize students as part of the partnership in the diverse engagement of multiple stakeholders.

Institutional-Level Evidence of Student Learning
In order to provide evidence of student learning, institutions use multiple assessment measures at multiple levels. Institutions are trending towards greater use of authentic measures of student learning such as rubrics, classroom-

VIETNAM JOURNAL OF EDUCATION
 57  based performance assessments and capstones to yield actionable information for the university (Jankowski et al., 2018). Also, the VALUE rubric is a common tool to assess general education programs (McConnell & Rhodes, 2017). Regarding program learning outcomes, in addition to standardized exams required by specialized accreditation, academic programs tend to use authentic assessment to engage meaningful discussion from faculty and students (Jankowski et al., 2018;Russell & Markle, 2017).
Communication of assessment results is a significant factor in building IQA culture (Fuller & Skidmore, 2014;Ndoye & Parker, 2010;Robinson et al., 2019). To communicate evidence of student learning, institutions normally share IQA reports internally and externally. Institutions can choose to provide evidence of student learning on their website or use the NILOA framework to demonstrate evidence of student learning (NILOA, 2019). In the Provost survey conducted by Jankowski et al. (2018), the researchers found out that how information was shared was more important than what was shared. They suggested providing more context, such as alignment with the institutional mission and student characteristics and interpretation of results so that general readers can understand and be interested in knowing about the topic. Jankowski and Provezis (2011) determined that institutions in their research provided more synthesis and multipleyear IQA results than long tables of data that were hard for internal and external stakeholders to read or comprehend.

Use of Institutional-Level Evidence of Student Learning
Use of evidence of student learning to make changes for quality improvement serves as an important factor in building the IQA culture (Kuh el al., 2015;Fuller & Skidmore, 2014;Meredith, 2013). Still, most universities researched in Jankowski et al. (2018) used the results of assessment of student learning to provide IQA reports for regional and program accreditation. At the institutional level, sample actions from the assessment results were changes to institutional IQA policy, placement for developmental courses, course prerequisite policies, program review processes, advising processes and general education and resource allocation. Most changes implemented from the assessment results were those at the course and program level such as eliminating redundant courses, changing course sequencing, aligning outcomes and addressing complex learning outcomes.

Reflection and Growth/Improvement Plan
In order for institutions to successfully reflect on assessment results and make improvement plans for their processes, they must build an IQA culture (Baham, 2019;Grandinetti, 2019;Verzinski et al., 2019;Robinson et al., 2019) and provide evidence of continuously closing the loop to ensure actions implemented had the desired impact on student learning (Kuh & Ikenberry, 2018). Fulcher & Bashkov (2012) suggested using the IQA participation rate, IQA quality and impact of the quality assurance (QA) office as three direct assessment measures. In order to evaluate IQA quality, institutions tend to build a meta-assessment process (assessing the assessment) and recruit faculty to review the IQA report (Fulcher et al., 2016;Schoepp & Tezcan-Unal;2016;Rodgers et al., 2012). Institutions also use indirect assessment measures such as local assessment surveys and interviews with faculty (Guetterman & Mitchell, 2015) or available assessment surveys about the culture of assessment (Fuller et al., 2015) to gain faculty feedback about IQA activities (Grandinetti, 2019; Robinson et al., 2017). The multiple assessment measures provide institutions triangulated data to evaluate the strengths and areas for improvement in the assessment process. Participants in the study included academic staff with additional roles such as program coordinator and director, quality assurance professionals and university administrators (specifically vice provost in academic affairs) working in U.S. higher education institutions. After receiving IRB approval, the researcher sent a recruitment email with a survey link in Qualtrics to about 2000 members in the ASSESS Listserv. This listserv is maintained by the University of Kentucky (UK) and managed by the UK College of Education to promote dialogue among assessment professionals, and provide information about developments in the field, upcoming conferences, job opportunities, calls for papers, etc. A statement of informed consent was included in the first question of the online survey, describing how the responses would be de-identified, stored and secured. The participants received three emails for three weeks as a reminder to complete the survey in Spring 2019. The total responses were 61; seven responses were deleted due to missing information. Due to the low number of participants from for-profit private institutions (N=6), the research excluded them for the data analysis. Table 1 shows the participant demographics.

Survey Instrument
The survey instrument was designed using the EIA rubric (2019 version). Four sections and corresponding dimensions were utilized to create statements. A four-point Likert scale was utilized for participant's responses the following question: "To what extent do you think your institution is doing the following:" The survey included four sections (12 out of 33 statements) of the EIA rubric to limit the time needed to complete the survey and to increase participation rates. In addition, the researcher structured the survey to distinguish it from two surveys that had been previously administered to the same ASSESS listserve: IQA culture survey by  and repeated questions from the Provost survey by NILOA (Jankowski et al. 2018).
In addition to the EIA rubric questions, respondents were asked to answer questions about their role, Carnegie classification, source of funding, and regional accreditation affiliation. Because the EIA rubric is designed for qualitative use, it has not been statistically tested for reliability. However, the researcher analyzed overall reliability and internal consistency for four dimensions to ensure reliability.
To analyze differences among more than two groups in several continuous variables, two-way MANOVA was the major measure to analyze the data. This study has two independent variables (source of funding and Carnegie Classification) with two levels (public and non-profit private; Doctoral and Master) and four dependent variables (diverse stakeholders' engagement, evidence of student learning, use of evidence of student learning and reflection and growth/improvement plans). G*Power is a tool to compute statistical power analyses and effect sizes. A power analysis for MANOVA in G*Power (Faul et al., 2009) was conducted for effect size. The research indicated a minimum of 34 participants to get a .95 power and .25 effect size with 17 participants per group.

Reliability
Reliability for all 12 test items was checked and the overall Cronbach alpha was .92. The researcher also checked the reliability of four subsections and the Cronbach's alpha was .50 for diverse stakeholders' engagement, r=.83 for evidence of student learning, r=.85 for use of evidence of student learning and r=.91 for growth/improvement plan. Since the reliability of "groups and individuals engaged in assessment activities" (r=.50) was lower than the generally-accepted .70 thresholds, the researcher decided to exclude it from the dependent variables in the MANOVA analysis and kept three dependent variables (evidence of student learning, use of evidence of student learning, and reflection and growth/improvement plan).

Analysis
A two-way multivariate analysis of variance was run to determine if evidence of student learning, use of evidence of student learning and reflection and growth/improvement plans differ by source of funding (public and non-profit private) and Carnegie classification (Doctoral and Master). Preliminary assumption checking revealed that the data were normally distributed, as assessed by Shapiro-Wilk test (p > .05). There were no univariate outliers as assessed by boxplot and Z-scores and no multivariate outliers as assessed by Mahalanobis distance (p > .001). There were linear relationships, as assessed by scatterplot, no multicollinearity as assessed by Pearson correlation (r = .712 and
The research failed to reject the null hypothesis; therefore, there was no difference in evidence of student learning, use of evidence of student learning and growth/improvement plans in source of funding for public and nonprofit private in Doctoral/Research and Comprehensive/Master institutions. So this would suggest that the operative "coercion" or "incentivization" associated with funding streams is not related to the participants' responses on the EIA Rubric sections in this study. See Table 2 for the extent the institutions scored in four dimensions of EIA rubric.

Discussions
Despite an important institutional characteristic (funding source), it is important to note that in the U.S., public, forprofit private and non-profit private HEIs are eligible for federal and state funding. Federal student financial aid (Title IV funding) is available to students at each of these three institutional types, and with this funding comes significant regulatory control (Jankowski & Provezis, 2011). In addition, the regional accrediting bodies carry out a "quasigovernmental" function in ensuring that accredited HEIs comply with federal regulations related to the receipt of Title IV and other federal funding. Thus, the underlying assumption that the categorical classification of funding sources influences the level of "coercion" or "incentivization" is obfuscated by the infusion of governmental funding in each of these HEI. The resulting regulatory requirements that "coerce" or "incentivize" institutional behaviors related to assessment and use of assessment findings for improvement (Gaston, 2014). In a different context, such as other HEI systems in non-U.S. contexts, the source of funding might be a clearly distinguishing institutional characteristic and the analyses might support the assumption that funding source is related to "coercion" or "incentivization".
Since the three source of funding categories are not distinguished so much in actual funding percentages due to the 90/10% rule pertains to the for profits, not-for-profits, and non-profitsno more than 90% of their funding may come from Federal sources (typically Title IV financial aid distributed to qualified students and returned in tuition and fee payments). The researcher looked closer at the Carnegie Classification between Doctoral and Master institutions due to their less dependence on public funding. Many large Doctoral and Master publics receive from 10% to 50% of their overall funding from state or Federal sources. Thus, there are some public institutions that claim to be "state located" because they are minimally "state funded", such as University of Michigan, University of Colorado, UT Southwestern, etc. So the distinctions specified for institutional classifications, especially the Carnegie "Very High" institutions which are independent of the intended policy coercion regardless of whether they are public or private in their control revealed the U.S. HEIs efforts to build its IQA system and culture following EIA framework for quality improvement (Jankowski et al., 2019;Kuh et al., 2015).
Although the research findings indicated the coerces or incentivizes funding policy had little impact on institutions' IQA system, federal and state regulation on all institutions that accept Title IV funding and get accredited, which coerces or incentivizes assessment practices and use for improvement might be true. Almost 35 states in U.S. use the performance-based funding model to allocate resources (Gaston, 2014). For example, Missouri performance funding model for public institutions required four measures in the model (student success and progress, efficiency and affordability, graduate outcomes, and comparator group) and evidence of student improvement on general education,

VIETNAM JOURNAL OF EDUCATION
 60  discipline assessments and licensure tests is an indicator for the transparency and accountability. Also, with the specific requirement on quality improvement in IQA system from regional accreditation standards such as HLC standard 4B, federal and state regulation would still have certain impact on these institutions' IQA practices.

Recommendations for NILOA EIA rubric
The overall reliability and internal consistency reliability in three sections examined via the survey indicated the EIA rubric is appropriate for use in the EIA evaluation and designation process. However, the section entitled "groups and individuals engaged in assessment activities" included two questions addressing two different topics (engagement of internal and external stakeholders); therefore, it is not surprising that the Cronbach alpha (r=.50) was lower than the accepted threshold. This finding suggests that NILOA should revise the items in this section to improve internal consistency reliability. Also, "groups and individuals engaged in assessment activities" included students as a stakeholder in IQA engagement, but Curtis (2018) argued that the EIA rubric did not indicate clearly the highest level of student engagement as partnership. Following the recommendation of Cook-Sather et al. (2014), student partnership in the IQA process might include allowing students to revise the institutional/program student learning outcomes (SLOs), reviewing the curriculum map, recruiting students to prepare IQA reports with program coordinators and participating in faculty meetings to discuss IQA results and suggest actions for improvement.
To close the loop, EIA rubric in Reflection and Growth/Improvement Plan sections suggested institutions should identify the strengths and weaknesses and make plans to make improvement next year. In addition to using a local IQA survey or IQA culture survey (Rodgers et al., 2016;Guetterman & Mitchell, 2015;Fuller et al., 2015;Fulcher & Jurich, 2012), the EIA can help U.S. and non-U.S. QA professionals to evaluate quality assurance processes, prepare a strategic plan of IQA activities and provide evidence for leaders to allocate appropriate resources to support IQA activities and engage diverse stakeholders to discuss and close the IQA loop. As mentioned above, responses were omitted from this analysis because, in the EIA Rubric dimension called "reflection and growth/improvement plans", seven participants did not provide responses one or more of the three sections and three participants provided no information on this dimension or the three sections. The researcher followed up via email with these participants and their responses indicated that they were confused when answering this section. Although this dimension had high internal consistency reliability, NILOA should consider revising the language or providing more context.

Implications for Vietnamese Higher Education Institutions
Internal quality assurance received much attention from higher education administrators since most HEIs build up to meet the external requirement of accreditation. However, during the implementation, there are always certain challenges in the implementation process such as additional workload, insufficient institutional resources to IQA activities and little impact on teaching and learning. In addition, best practice of IQA always demonstrate the institutions' efforts toward continuous quality improvement. Therefore; it is necessary for Vietnamese HEIs to assess the IQA system. EIA framework with eight dimensions serves a good IQA framework for Vietnamese HEIs to benchmark its current IQA activities with international standards. Most importantly, this IQA framework supports leadership use of evidence of student learning for resource allocation (Dickson & Treml, 2019;Grandinetti, 2019;Verzinski et al., 2019) and externally benchmark with the current IQA system (Robinson et al., 2017).

Limitations
Three important limitations apply to this study. First, data were collected in a short timeframe (three weeks). Second, although the ASSESS Listserv includes about two thousand members, there was no way to know the exact number of members who were qualified for the survey as many do not engage directly in IQA process. Therefore, it was difficult to have an accurate percentage of response rate. Third, the EIA rubric was designed for qualitative narrative responses; consequently, statements in the survey are somewhat lengthy.
Although this research focused on four sections of the EIA rubric related to the IQA process, other sections and dimensions included in the rubric offer the opportunity for further study. In particular, multiple case studies have identified IQA plans, resources and leadership as significant factors in sustaining an IQA culture (Baham, 2019;Dickson & Treml, 2019;Grandinetti, 2019;Robinson et al., 2019;Verzinski et al., 2019). Future research should include all 33 statements from the EIA rubric with an extended period for responses. In addition, the reliability for the entire rubric should be tested. Given the low response rate, more targeted recruitment efforts might increase response rates from those with specific job titles related to IQA (Jankowski et al., 2018;Fuller & Skidmore, 2014).