Accreditation Compliance Institutional Effectiveness Index

Assessment of Extension and Engagement Activities Summary

Introduction

As a land-grant institution, NC State University has had a significant extension component as a part of its rich history.  The prominence of extension and engagement has been amplified with the creation of a vice chancellor position and the July 2001 hiring of a highly qualified professional, Dr. Steve Jones.  Under his leadership, the visibility of extension and engagement within the campus community and its importance to the various external constituents served have already been enhanced.  Extension, coupled with terms such as outreach, public service and now engagement, has in general never been as clearly definable or as easily measurable as research and teaching.  Also, extension is so multi-faceted and diverse that efforts to evaluate its quality, document its scope, and assess its impacts have lacked standardization and uniformity across the disciplines.  The evaluation and assessment processes employed by the respective units have been relevant to their planning, budgetary, and operational needs.  Through the Compact Planning process, a more comprehensive approach to the development, delivery, documentation and assessment of these activities will be accomplished. As a result, evaluation and assessment will become even more entrenched into the overall campus extension culture.  The Vice Chancellor will not mandate a specific methodology, but rather provide an environment that supports and encourages unit-based assessment and its utilization in the planning cycle.  Ultimately, it is anticipated that all colleges will elevate the status of extension and engagement and that professional achievements in this arena will be more highly valued in the University's reward and tenure process.  NC State will be better positioned to fulfill its mission and to improve its image as the "peoples' university." 

Mission

Extension and Engagement at NC State University provides leadership and vision for University partnerships with external communities that facilitate research and discovery; teaching and learning; and outreach and service.  As the state's only research university in the land-grant tradition, NC State has a unique mission to serve the citizens of North Carolina through technical assistance, professional development, lifelong education, technology transfer, and other means of applying knowledge to real world issues and problems.  Faculty, students, and staff from every academic college engage in collaborative research, learning, and service partnerships with business, industry, government, and communities in the Triangle region and across the state.  Extension and engagement imperatives include economic development, environmental stewardship, K-12 education excellence, and leadership development.  Personnel and activities within extension and engagement are present in every one of North Carolina's 100 counties plus the Cherokee Indian Reservation, reaching more than one million citizens annually.  As part of the University's land grant mission, extension as presented in this report illustrates the breadth of these customer-driven activities that comprise the campus-wide efforts.

Background

Given the diversification that characterizes University extension and engagement, the spectrum of activities as well as their respective impacts must be reviewed on a case-by-case basis.  As an example, many of the extension and engagement activities are one-on-one consultations with small business owners to devise more effective business, marketing and development plans for future growth; in-plant assessment of manufacturing processes and operations to improve efficiencies through the integration of new technologies; program planning sessions with advisory committees; development of customized, contractual training modules based on the needs of a company's workforce to improve profitability and stimulate economic development.  The recently established Economic Development Partnership Office will devise prediction models and perform post-initiative assessments as a fundamental component of its overall function. Even though all campus extension units have evaluation and assessment processes in place, there are varying degrees of sophistication that range from very structured and formal to less structured yet practical.  The processes have been developed and owned by the individual units to best accomplish their intended purposes and to sustain their operations.  The assessment of extension and engagement efforts cannot, therefore, be judged solely by formal outcome measures or even the utilization of the more traditional means of end-of-program evaluations and customer satisfaction surveys.  In those instances where financial support includes appropriated dollars and grants, documentation and accountability standards are very specific and assessment procedures strictly monitored.  For self-supporting extension and engagement units, an informal and less structured system often proves most feasible.  A simple yet very significant impact measure applicable to these units would be the volume of repeat customers.  Also, the low number of registration cancellations and the length of contractual relationships would be other indicators of customer satisfaction.  Through a mix of such techniques and strategies, extension and engagement provides access to learning opportunities and to the resources of NC State with evidence of its impacts on individuals, organizations, businesses, industries and communities.

Report Preparation

The background information and support documentation for this report were secured by the leadership team of the McKimmon Center for Extension and Continuing Education (MCE&CE).  They conducted the interviews and assimilated the core materials used in generating this report by working primarily with the members of the Extension Operations Council (EOC). The Council's membership is comprised of a representative from each of the ten colleges at NC State plus the Vice Chancellor and Assistant Vice Chancellor of the Office of Extension and Engagement.  An email message was sent to each member of the EOC requesting information, assessment processes, evaluation instruments, assessment tools, impact results and other relevant reports and documents.  Web-based copies are available via the Document Index that accompanies this report. After the draft report was composed, each EOC member was asked to review the document, make suggestions for corrections, and provide additional information to better reflect their individual college's activities.

Unit-Based Extension and Engagement Evaluation and Assessment

To depict the varying levels of assessment that exist within the campus environment, the following information is presented to characterize each of our ten colleges plus one centralized extension and continuing education unit.

College of Agriculture and Life Sciences:

The primary extension component of CALS is the Cooperative Extension Service.  CES programs have been recognized as outstanding given their assessment, evaluation, and impact processes.  CES utilizes a large variety of assessment methods:  end-of-program evaluations, one-on-one interviews with participants, case studies, participant testimonials, on-farm demonstrations, success stories, cost benefit analysis, and the Advisory Leadership System.  The primary program areas of CES include agricultural, forest and food systems; conserving and improving the environment and natural resources; building quality communities; strengthening and sustaining families; and developing responsible youth.  An example of one of 20 CES major programs, Agriculture and the Environment, was created to develop and implement educational programs to improve clientele understanding of the complex relationships between agriculture and the environment.  CES provides most of the curriculum and training for these programs.  This major program, including education, training and demonstrations, helps extension's customers make sound decisions that protect the environment and maintain productivity through adoption of sound practices to manage water, wastes, soil, nutrients and pesticides.  The impacts of this program were reported statewide; i.e., 4,279 acres were established in new wildlife habitats; 172,500 pesticide containers were collected and recycled; 2,439 waste operators or applicators were certified to manage waste systems; 3,190 nursery, greenhouse and turf professionals were certified; and 19,188 pesticide applicators were certified or re-certified.  The impacts of this program illustrate how training, education, and assessments resulted in dramatic decision-making and improvements that enhanced communities and the quality of life for North Carolinians. For example, follow-up contact with participants in CES's Animal Production and Marketing Systems programs indicates that many go on to adopt farm management practices that increase their income.  Altogether, CES programs involved 93,101 volunteers with an in-kind value of $9,028,297.  Through its programs, CES programs produced 7,280,012 known teaching contacts with the state's citizens.  Because CES receives federal funds, the evaluation process is a requirement. The U.S. Department of Agriculture has listed the CES Annual Report on its website as an exemplary model for other states to review and to use as a benchmark for their assessment processes. 

College of Design:

The college does not have a formally structured process for evaluation of extension activities.  Many of the college's "outreach" efforts are actually service learning projects undertaken by students in their design studio courses.  Although there is a long record of project successes, no systematic evaluation process has been followed and documented.  Currently, the college uses individual faculty annual reports and yearly assessments to collect data.  Examples of these efforts are the revitalization project in Halifax County and signage for the International Special Olympics.  The responsibility for organizing the college's extension activities has been most recently assigned to a faculty member who will fully incorporate assessment/evaluation processes into future relevant activities or programs that the college delivers.

College of Education:

The College of Education extension effort is not centralized, but rather consists of numerous projects and centers that operate with relative autonomy.  Highlights include these:

Consistent with the college-wide emphasis on teaching and learning with technology, there are more than 20 web-based extension projects operated by faculty and staff of the college for links to some of these "Online Tools for Schools".

ˇ        Significant partnerships include the Technology Assessment Project for North Carolina Teachers in conjunction with UNC-Greensboro and UNC-Chapel Hill.  This site provides information about the state-mandated technology requirements and assessment, and also provides resources for learning the required skills. 

ˇ        The Center for Research in Mathematics and Science Education is the only research center in the state-wide Mathematics and Science Education Network, and the home of the MSEN PreCollege Program, which enrolls 450 middle grades and high school students in year-round enrichment activities.  The mission of this program is to keep underrepresented students on a college-bound track leading to Science, Technology, Engineering, and Mathematics careers. 

The extent of systematic evaluation characterizing these efforts is substantially varied, ranging from simply recording hits on a web site to rigorous, formal evaluation research producing articles published in professional journals.  The typical evaluation of on-site extension efforts not part of externally funded projects (e.g., workshops, demonstrations) consists primarily of output data such as the number of workshops taught, the number of teachers in the workshops, and the number of students served by these teachers attending the workshops.  There is a Faculty Activity Report that is filed with the Dean's Office twice each year to provide documentation of the outreach activity that has occurred between the faculty members and a school district or teacher(s).  This report, however, does not evaluate the impact or outcome of the activity being reported.

College of Engineering:

The Industrial Extension Service is the primary extension component of COE.  IES serves clients within the state of North Carolina in the areas of continuing professional education and direct technical services.  In the manufacturing community alone, there are approximately 12,000 companies within the borders of the state.  IES tracks the numbers of clients they are able to serve and determined, in early 2000, that they had reached a plateau in the numbers of companies that could be effectively served.  The existing IES staff could only serve approximately three percent of the total North Carolina base.  The field staff group was charged with becoming acquainted with potential clients, evaluating their needs, developing a program to satisfy them, and actually delivering the services to the client companies.  This methodology limited the number of companies that could be effectively served.  After study, it was determined that a major reorganization was needed to extend the IES reach to better fulfill our mission.

IES underwent a reorganization from a silo organization, with several distinct business units, to a functional organization that is able to easily draw on all the organizational resources.  Based on existing strengths, the organization was realigned into eight functional groups.  These new groups covered the basic IES activities of business operations, client contact and needs assessments, product delivery, marketing and public relations, and the development of new business opportunities.

On July 1, 2000, IES was formally structured into the areas of Business Operations; Business Development; Marketing and Public Relations; Environmental, Safety, and Health (EHS); Advanced Performance and Standards (APS); Lean Enterprise Advancement Program (LEAP); Energy Programs; and the Environmental and Technology Group (ETG).

Functions that had existed in multiple sections of the prior organization were consolidated under operating teams.  For example, the functions of events management, accounting and finance, general administration, information technology, human resources, and contracts administration were pulled together to form the Business Operations group.

The client contact arm of the organization was re-structured into five areas.  To accomplish this change, the field staff was split into groups that had primary responsibility for direct client contact and needs assessment and others that had primary responsibility to deliver the actual services to the clients.  Each of the delivery groups has responsibility for product delivery, product pricing, and initiating marketing materials.

During the reorganization process, new staff members were added.  The new staff are generally deployed across the state.  This deployment, an opening of eight new offices, fits with the philosophy of reaching more of the North Carolina companies as well as the goal of locating resources in a manner convenient to the client base.

As a result of the reorganization process, IES has been able to eliminate three levels of management and also maintain a seamless service delivery to our clients.

The Industrial Extension Service is a dynamic programming and client services unit with detailed assessments, surveys, evaluations, and impact data.  A board of industry experts serves as an advisory committee providing valuable input to the types of programs and services that are offered by IES.  With federal funds to support many of the IES programs, detailed assessment reports are a requirement.  For example, IES conducts three evaluations for most of its programs.  Immediately following a program, an end-of-course evaluation is distributed to program participants.  Approximately 30 days after the program, another customer evaluation is disseminated for completion.  This survey evaluates customer satisfaction and solicits business needs and new contacts.  Approximately 9 - 12 months later, a research firm, under contract with the national Manufacturing Extension Partnership (MEP), conducts an independent study by reviewing the customer quality satisfaction evaluations of the IES clients and program participants.  The study includes penetrating questions aimed at quantifying the impact of the project or program.  The results from this study are used to generate IES's economic impact statements which are used in annual reports, special promotion campaigns, and is reflected on their website.   The "Success Stories" for each Business Solution on the IES site give examples of projects, follow-ups, and impacts.

Another extension component through the Industrial Extension Service is the North Carolina Solar Center.  NCSC promotes sustainable energy solutions by accelerating the use of solar and renewable energy technologies through education; research; technical consultation; business assistance; and program development.  The success of NCSC's programs is measured by utilizing end-of-program evaluation forms and follow-up surveys to past participants.  The forms and surveys indicate the number of participants in workshops, the number of recipients receiving technical assistance, and the number of grid connected PV's or separate PV installations, wind, and hydropower systems installed in North Carolina.

College of Humanities and Social Sciences:

Humanities Extension has enjoyed long term partnerships with the Department of Public Instruction (DPI) and the Library of the State of North Carolina.  The programs presented have been for teachers in the public school systems and librarians across the state.  The evaluations utilized have ranged from very general in nature to more complex in the types of questions presented to the participants.  The basic evaluation form asks for the typical customer satisfaction feedback.  The next level requests feedback as to how the program is related to or would create an impact on the participant's job.  The more complex evaluation form, distributed for the "NC Reads NC: Our Poets Speak" program, includes interim and final participant responses as well as an assessment from the host librarian.  The most distinctive fact related to the partnerships with DPI and the State Library and the evaluation process is that the evaluation forms are developed and administered by the client.  By the clients requesting additional programs and given the long-term nature of the partnerships, the positive gains from participation are clearly evident.

College of Management:

The Executive Education Program at the college has been evolving over the last several years with programs offered that have been both open enrollment and customized.   According to the program representative, the typical end-of-program evaluation forms have been utilized and summarized.  The feedback presented in the summaries has been critical in the planning phase related to topics and presenters for subsequent offerings.  For example, a program was customized for Blue Cross Blue Shield (BCBS).  The evaluation form requested the typical feedback, although the unique aspect of the assessment was that Blue Cross Blue Shield  employees were asked to suggest changes that they would make for a different audience, i.e., middle and senior management.  Such feedback has been used in future program development.

College of Natural Resources:

The college has active extension and engagement programs in each of the three departments:  Forestry, Wood Products, and Parks, Recreation and Tourism Management.  Virtually every aspect of the college's extension programs from objectives to assessment is rigidly monitored and planned.  A variety of evaluation techniques are employed to assess extension efforts.  Examples are surveys, pre- and post-testing, testimonials, one-on-one consultations, and workshop surveys/evaluation forms.  Additionally, the county agents carefully track all activities in their counties, reporting these activities and the impacts back to the University through the Extension Reporting Systems (ERS).  Of the various forestry extension programs that have been offered by the college, the Forestry Issues 2001 - 2002 Teleconference Series are presented as the example.  Nine teleconferences were held once a month from September 2001 - May, 2002.  The broadcast, initiated from Biltmore Hall, included eight sites across the state.  Compared to the teleconferences presented during 2000 - 2001, an increase of 23.87% was reflected in the total registrations (737 registrations).  Approximately 2,294 continuing forestry education (CFE) credits were awarded.  The series resulted in an estimated $428,748 in total cost savings when compared to attending traditional conferences.  Participants felt the speakers did a very good job presenting the information.  The participants also perceived that the teleconference technology had little to no impact on their ability to interact with speakers; had improved the quality of information they received; and increased the level of satisfaction with the teleconference series.  Participants' overall level of satisfaction with the teleconference series ranged from satisfied to very satisfied.  A review of the evaluation/impact summaries indicated a substantial cost savings and convenience factor of presenting the series via a teleconference format versus the face-to-face delivery method.

College of Physical and Mathematical Sciences:  

The primary extension component of PAMS is The Science House.  According to the Annual Report for 2001 - 2002, The Science House is a unique enterprise: a nationally recognized model of university involvement with schools.  Each year The Science House brings science learning to more schools and teachers and helps to encourage a new generation into careers in science, mathematics, and technology.  The effectiveness of its programs is due to the commitment from the college as well as to the private, corporate and governmental organizations that are partners and funders.  Due to the fact that The Science House receives a large number of federal grants from NSF, NASA, NOAA, and the U.S. Department of Education, as well as other sources such as the Burroughs Wellcome Fund, and the Howard Hughes Medical Institute, the evaluation process for its programs is very complex and detailed.  Formally stated goals and purposes are included in the programs.  Many of the evaluation summaries give references to evaluative results and impacts.  The evaluations cover the spectrum from "feel good" anecdotes to more formal efforts.  In many cases, the program feedback has led directly to the creation of new products/services in addition to the refinement of existing products and services.  Several of the programs presented by The Science House are Science and Mathematics Colloquies, the Imhotep Academy, Student Camps, the HHMI (Howard Hughes Medical Institute) Precollege Collaboratory, Science on the Road, Teacher Training Workshops, and K-12 Outreach from the NSF Fungal Genomics Laboratory.  An example is the Science and Mathematics Colloquies.  The program targets high-achieving high school students to introduce them to science as an activity and to show them the vital, interdisciplinary nature of modern science.  Each colloquium consists of ten afternoon sessions during which approximately 20 participants learn about a topic of current interest.  In its fifth year of offerings, the student evaluations are a testimonial to its effect on their choices regarding science and mathematics.  To illustrate, one student stated "it made me enroll in more science classes next year because I know I want to be working in the science field."  Another student stated,  "I used to think that weather was not important!  After this class I want to know why everything occurs."  According to the evaluation report, over 85% of the students participating hoped to pursue a career in science and mathematics. 

College of Textiles:

The programs offered by the Textiles Extension Office have been recognized as outstanding programs as well as leaders in the field of textiles with extension/outreach efforts.  The evaluation form most often used with the textile short courses is the typical end-of-program/customer satisfaction survey.  The example presented is a training assessment with the Short Staple Spinning in-plant offering with Eden Yarns.  The design of the project was aimed at very specific assessment goals.  The final assessment results were not as originally planned due to a variety of reasons (money for funding the assessment, requirement of a great deal of time and resources, lack of company input), but revisions were made to the initial plans to produce   valuable participant responses.  Over a 17-month period of time, participants from the open enrollment Spun Yarn Manufacturing Program were surveyed.  Results were used to refocus class material to enhance the learning impact of future classes.  At the completion of the assessment, the textile extension personnel felt they had learned some lessons that could be shared with other outreach/extension/engagement units regarding the difficulties in trying to conduct a research assessment out in the "real world."  Their project yielded examples of what can go wrong when trying to conduct assessments in the business environment versus the typical university classroom.

College of Veterinary Medicine:

The college has a very active extension program with many programs and conferences.  The programs and conferences are developed in response to pre-determined educational needs and interests and, therefore, provide high-quality continuing education opportunities at the graduate (post DVM) and specialist level, as well as to

non-veterinarians.  The programs use multiple technologies to reach diverse audiences and take the form of short courses; local, state and international conferences; hands-on laboratories, and in-house training courses, depending on the educational needs of the targeted audience. The methods of evaluation range from the traditional end-of-program evaluation to complex, very detailed assessments.  Completed evaluation forms are reviewed by the program organizers and then entered into the computer from which summaries are generated and shared with the faculty members involved with the program.  Also, after each program, the faculty/instructors and the extension staff meet to discuss and evaluate the different aspects of the course, to discuss how improvements can be implemented to make the course more informative and how to disseminate the information more effectively to the participants. External presenters to the college are sent letters of acknowledgement, reflecting their scores, along with any comments directed specifically to the presenters.  Suggested changes to the program, topics, presenters, etc. are implemented in the next offering based on feedback from the evaluations.

McKimmon Center for Extension and Continuing Education:

Consistent with NC State University's land-grant mission, MCE&CE provides access to learning opportunities that enable citizens to continue their education, advance their careers and enrich their lives.  Relevant technical assistance is provided to and applied research performed in collaboration with schools, community colleges, municipalities and public agencies.  For more details, go to MCE&CE website and select "Annual Report."  The major noncredit training component, Continuing and Professional Education (CPE), serves more than 165,000 participants in seminars, conferences, and workshops.  The most common assessment is the end-of-course evaluation or customer satisfaction survey.  Feedback from evaluations has a direct impact on the program development for the next offering.  Additional topics are identified as well as new instructors or presenters are integrated into the curriculum enhancements.  Program advisory committees and faculty representation are critical in the development of new programs or in the revisions of existing ones.  Email surveys have now been implemented to gather impact results from participants in diploma/certification based offerings after a period of 60 - 90 days has lapsed since the program offering.  Pre- and post-assessments are conducted on a regular basis with the CPE programs to determine the level of participant knowledge and understanding prior to and at the conclusion of the program.  Evaluations for the facilities and facility support also play a vital role in the planning process for the next program offering.   The volume of repeat customers and the minimal number of refunds are impact measures that reflect an overall rate of customer satisfaction.  The Encore Center for Lifelong Enrichment conducts one-on-one surveys to obtain feedback from their membership for programming and level of membership satisfaction.  All pertinent feedback gathered through these various means are integrated into future program planning.  The Center for Urban Affairs and Community Services (CUACS) and the Department of Public Instruction continue a relationship that started approximately 12 years ago that encompasses end-of-year and end-of-grade testing and assessment for all public middle school students.  The length of the partnership based on contractual funds implies that the customer service and satisfaction continues to be maintained by CUACS at an exemplary level.

Conclusion 

The extension and engagement activities at NC State University are quite diversified, but at the same time, are integral to our overall land-grant mission. Evaluation and assessment throughout extension and engagement reflect the general University accountability climate and will become increasingly embedded in the Office of Extension and Engagement's planning and  continuous improvement process as the University moves into the 21st Century.

The Document Index accompanying this report has additional information about the units' assessment of extension and engagement activities.                                                         

            3/25/03