Description and Evaluation of a Prenatal Exercise Program for Urban Aboriginal Women
Upon reading the title of the program case study, “Description and
Evaluation of a Prenatal Exercise Program for Urban Aboriginal Women,” the
Stake’s Countenance Model immediately came to mind as a possible model to evaluate this program. The two key components or countenances of Stake’s evaluation model
are description and judgement or evaluation, so it seemed like a logical fit as
both components appear in the title. However,
after careful reading of the text, it is apparent that this paper is more descriptive
about a variety of its program components and less evaluative. Another possible evaluation
process that would be effective in addressing this particular program is the
Utilization-Focused Evaluation or U-FE.
What is Utilization-Focused Evaluation?
Foundational to U-FE is that an evaluation and its findings should
focus on the “…intended use by intended users” (Patton, 2002, p.1). Utilization-focused evaluation “…is a process
for making decisions about issues in collaboration with an identified group of
primary users focusing on their intended uses of evaluation” (Patton, 2002,
p.1). From the beginning, the evaluator
or facilitator works with the primary users to plan an appropriate evaluation
based on the nature and situation of the program. Through this ongoing interactive process, the
evaluator and primary users collaborate to determine the following components
of evaluation:
- Purpose – formative, summative,
process
- Data collection – qualitative,
quantitative, mixed
- Design – experimental,
naturalistic, quasi-experimental
- Focus – inputs, outputs, outcomes,
cost-benefit
This evaluation
process is not static, but instead, is based on situational responsiveness which
guides U-FE. Another key component of
U-FE is that it “answers the question of whose values will frame the evaluation
by working with clearly identified, primary intended users who have
responsibility to apply evaluation findings and implement recommendations”
(Patton, 2002, p. 1). Utilization-focused
evaluation takes a constructivist approach to evaluation as primary users build
their understanding of the process and use of evaluation. By actively being involved in the process,
primary users are more likely to take ownership for the evaluation and implement
its findings. Although U-FE is based on
collaborative and constructivist learning, the evaluation process is framed by
a twelve part checklist that is organized according to the primary tasks of
evaluation and the challenges identified for each task.
Why use U-FE to evaluate the Prenatal Exercise
Program for Urban Aboriginal?
Utilization-focused evaluation would
be an effective process for evaluating the Prenatal Exercise Program for Urban
Aboriginal Women. In the discussion section of the paper, the
authors suggest that continuance of the prenatal exercise program may have
occurred ``had it been a `grassroots` initiative or designated as having priority
status by Aboriginal leaders″ (Klomp, Dyck, & Sheppard, 2003, p. 237). Program evaluation
needs to be sensitive to cultural values and norms and in this particular
program, Aboriginal values and norms. In
U-FE, the intended users are more likely to enact the recommendations if they know
that their values and norms are the framework for the process. This evaluation process would provide a
collaborative working relationship between the facilitator and the primary
users of the findings and recommendations to be implemented. Primary users, as identified by the U-FE
checklist, include people who have a direct stake in the evaluation and meet
identified criteria (criteria are negotiable). With this particular program, the stakeholders
are not clearly identified, but primary users for this evaluation could include
the Aboriginal Project Facilitator, the Elder, a representative from the National
Health Research and Development Program (NHRDP), a representative from the
YMCA, a health representative such as the Registered Nurse Coordinator, and the
authors of the paper, Klomp, Dyck and Sheppard as they developed the program. Although it may be a challenge to coordinate
U-FE, the detailed checklist provides a framework for constructing an
evaluation including identification of intended users, and role clarification for
the facilitator and intended users.
The U-FE checklist is intended to
guide the facilitator and the primary users to select an appropriate evaluation
purpose, data collection methods, design and focus that align with the context
and values of the primary users. To
begin this process, the facilitator and the primary users would work together
to establish a purpose and focus, and one method to accomplish this would be the
co-construction of a logic model. This
collaborative process would provide an overview of the program and identify the
inputs, outputs and outcomes as well as any assumptions. This process would also help to build
understanding of the evaluation process which is a main goal of U-FE, as well
as clarify the program for both the facilitator and the primary users. Although the description of the program
begins with an implied long-term objective, reduced gestational diabetes
mellitus (GDM) for urban Aboriginal women and ultimately prevention of type 2
diabetes in future generations, the purpose of the program is never explicitly stated. Some of the inputs (personnel, funding, materials, partners), outputs (45 minute fitness classes), and outcomes (medium term - improved level of fitness and self-esteem; long term - reduced GDM and type 2 diabetes) are mentioned, but not explicitly identified, so determining
if they are present, intended, or achieved is challenging.
However, the purpose of an evaluation is to provide an assessment of the
program and base that on the program information provided in whatever format or state
it is presented. This is why U-FE is an
appropriate process for evaluation of this program. The ongoing collaborative
process lends itself to developing a logic model to clarify the purpose and
focus of the evaluation alongside the primary users whose values remain at the
forefront of the process.
Once the purpose (formative,
summative or process) and focus (inputs, outputs, outcomes, or cost-benefit)
are identified, then data collection and evaluation design can be addressed. A possible document that could be reviewed by
the evaluator and primary users to support these next steps would be ``First
Nations Child and Family Caring Society (FNCFCS) of Canada Program Evaluation
Research: Final Report.″
In this March 2006 report, the evaluator reviewed
various evaluation resources that could be used or adapted to meet the needs of
the FNCFCS, First Nations programs, and general programs. One of the recommendations of the report was
a proposed evaluation framework, and a component of this framework included
that evaluation be utilization focused. This
report supports a utilization-focused evaluation as it identified the importance
of a participatory model, and of the evaluation meeting the needs of the
primary users. This report would provide
a framework for evaluation of the prenatal program as well as a data base for
accessing evaluation models that address First Nations programs as the report includes
features and limitations of the various models reviewed.
Utilization-Focused Evaluation is about a process of evaluation rather than a particular model. Although this approach to program evaluation could be costly in terms of time and money, a constructivist and collaborative inquiry based U-FE, framed by the values of the primary users, may be the most appropriate way to ensure intended use by intended users.
