TEI

Building Performance Measurement Capacity in Tribal Communities

Building Performance Measurement Capacity in Tribal Communities

Tribal Home Visiting Program grantees and communities are building a key evaluation capacity: performance measurement. Here are some of the benefits grantees have reported as a result of the performance measurement planning process.

ICON-dialog-2b“We learned to anticipate data needs and hypothesize about program impacts. The benchmark process required us to get concrete about program processes and outcomes.”

— Tribal home visiting grantee

Advancing Community Engagement Around Data

Performance measurement planning provides an opportunity for grantees to engage tribal communities and partners in discussions about data, often for the first time. Together, they consider what kind of data would best serve the community, how to use existing data, and how to coordinate data collection and share data across programs. These community-wide discussions helped grantees examine and strengthen their home visiting program’s relationship with other programs within the community’s early childhood system.

Tailoring Performance Measures to Community Needs and Strengths

Many grantees say the process of tailoring performance measures to fit their communities supports tribal sovereignty and self-determination. For example, in a community with an established goal of increasing the number of mothers who breastfeed, the grantee might choose to measure the percentage of mothers who initiate breastfeeding. Collecting data on activities and outcomes that are important to a tribal community results in information that is useful for tribal decision making.

Examining and Improving Programs

When grantees plan performance measurement, they examine processes across their entire program, including intake, screening and referral, and delivery of home visits. For example, the need to track screenings and timely referrals for performance measurement can lead to the development of coordinated screening and referral processes across multiple programs, with the goal of creating a more seamless experience for families. Collecting data on screenings and referrals over time can help programs see whether families are receiving these services as planned.

Building Capacity

TEI provides targeted technical assistance and tools to help grantees develop their performance measurement plans. Grantees that build their capacity to collect and use data are able to measure and improve performance despite disruptions in staffing or funding. These skills are transferrable to other programs and initiatives.


Performance Measurement in the Tribal Home Visiting Program

Performance Measurement in the Tribal Home Visiting Program

What Is Performance Measurement?

Performance measurement is the regular collection and reporting of information about program processes and outcomes in order to monitor and improve the quality of the program.

Why Measure Performance?

Many federal grant programs require performance measurement to understand how programs are operating and progressing. Performance measurement can be useful for local communities as well. Tribal Home Visiting Program grantees use performance measurement data to understand how well programs are meeting their families’ needs; track improvement over time; and communicate program successes and challenges to partner agencies, advisory groups, and tribal leadership.

ICON-dialog-2b“The program learned a lot from the performance measurement process…[and] will continue looking at the data in real time to identify gaps.”

— Tribal home visiting grantee

How Do Tribal Home Visiting Grantees Measure Performance?

Legislation requires Tribal Home Visiting Program grantees to establish performance measures so they can track, measure, and report improvement across six areas:

  1. Maternal, newborn and child health
  2. Child injuries, child abuse, neglect, or maltreatment, and reduction of emergency room visits
  3. School readiness and child academic achievement
  4. Crime or domestic violence
  5. Family economic self-sufficiency
  6. Coordination and referrals for other community resources and supports

Grantees funded between 2010 and 2015 developed and implemented individualized performance measurement plans for monitoring program performance in these areas. Each of these 6 areas included multiple items, totaling 36 constructs or indicators across the 6 measurement domains. As part of this process, grantees developed data collection and management protocols, analysis plans, and data systems capable of housing and linking data across programs.

Grantees funded from 2016 to present use a standardized set of 12 core performance measures and select from a set of 11 additional measures. The standardized measures were developed by the Administration for Children and Families with the input of tribal grantees and technical assistance providers. Learn more about the redesign of the Tribal Home Visiting Program Performance Measurement System.

How Does TEI Help?

The Tribal Evaluation Institute provides technical assistance on performance measurement to all Tribal Home Visiting Program grantees. TEI works with grantees on developing site-specific performance measurement plans, defining quantifiable and measurable indicators, supporting home visitors in collecting data, developing data collection policies and procedures, establishing data systems, monitoring data quality, conducting analysis, reporting, using the data to drive improvement, and sharing lessons learned about performance measurement.

Cover of the Tribal, Maternal, Infant and Early Childhood Home Visiting: A Report to Congress

Grantees have built their capacity to collect data, track performance and improve programs. Learn more about their experiences in the Tribal Home Visiting Program Report to Congress (PDF, 1.84mb).

Download report



Our Capacity-Building Approach

Our Capacity-Building Approach

Respect. TEI is dedicated to building relationships that help us understand grantees’ needs, contexts, and priorities. We respect tribal processes and decision making, and we recognize that each tribe, tribal organization, and grantee team has its own priorities and requirements.

Responsiveness. TEI understands that grantees must consider many factors when developing and implementing evaluations. Our responsive approach integrates cultural and scientific rigor, community interests and priorities, contextual realities and constraints, and grant requirements. We draw upon the knowledge of partners, consultants, and experts to guide our work with grantees.

Community engagement. TEI supports tribal and community input and recognizes grantees’ responsibilities to tribal entities. We know it takes time to gather critical input from sources such as tribal and agency leadership, community advisory groups, program stakeholders, and cultural advisors and elders.

Capacity building. TEI builds evaluation capacity on a foundation of shared understanding and trust. We believe learning is reciprocal. We facilitate the application of technical and methodological expertise and adult learning principles while providing accessible, responsive, and tailored guidance for each grantee.

MODEL OF TEI'S CAPACITY-BUILDING APPROACH

TEI Conceptual Model

The Tribal Home Visiting Evaluation Institute (TEI) is funded by the Office of Planning, Research and Evaluation (OPRE) within the Administration for Children and Families (ACF), Department of Health and Human Services under contract number HHSP2333201500114G. TEI2 was awarded to James Bell Associates, Inc., in partnership with the University of Colorado’s School of Public Health, Centers for American Indian and Alaska Native Health and Michigan Public Health Institute.


Plan-Do-Study-Act (PDSA)

Plan-Do-Study-Act (PDSA)

Plan-Do-Study-Act is an iterative, four-stage problem-solving model used for improving a process or carrying out change.

Step 1Step 6Step 7Step 8

Step-by-Step Walkthrough

detail of PDSA model

Plan

In Step 1, you will identify an opportunity for improvement for your PDSA cycle. Let data guide your selection of a topic. Data can be both numerical (like client or program records) and descriptive (like results from customer satisfaction surveys or general staff feedback). Also, you can start thinking about and securing the resources you will need for your PDSA cycle and getting any necessary approval to start the CQI project.

Ask yourself: What are some areas the program can improve on? What do the data tell us about how well we are doing in those areas?

In Step 2, you will  identify staff, program participants, community members, and partners who have knowledge of the targeted area so you can assemble your CQI team. The team should further discuss the opportunity for improvement and draft an initial SMART (Specific, Measurable, Achievable, Relevant, and Time-Bound) aim/goal statement. Teams often find it helpful at this point to assign roles to members (including leader/facilitator, note taker, document manager, and meeting scheduler), create a project timeline, and organize project plans into a summary document, called a team charter.

Ask Yourself: Who can contribute to our team’s PDSA cycle, and what will their role in our CQI work be? What is the goal of the PDSA cycle?

In Step 3, clarify how the program is currently operating in the area you have identified to improve with your PDSA cycle. Consider creating a process map or flow chart to examine how the program operates and look for data and information you can use to establish a baseline (i.e., starting point) to compare to when your PDSA cycle is complete. Before coming up with a solution, ask your team to first think about what is causing the problem to exist. You may find it helpful to create a cause and effect diagram (e.g. fishbone diagram) to explore the root cause of the problem. After creating a list of all the possible reasons the problem exists, ask the team to pick just one that you will try to change with your CQI work.

Ask Yourself: What do we know about the area we have chosen to improve? What is the main reason this problem exists?

Step 4 is when the team will brainstorm and collectively think about all the possible solutions to the root cause of the problem. Teams should look to their program model or existing best practices when possible. The team should select a potential solution within its scope of control that could best address the problem.

Ask Yourself: What could we do to improve the challenge we are experiencing?

In Step 5, your team should develop a theory for improvement, which is a prediction of what will happen when your team tests the potential solution on a small scale. Your team will also develop a strategy for how you will test your theory for improvement (e.g., who will help, what materials are needed, when it will occur).

Ask Yourself: What do we think will happen if we try out a specific potential solution? What do we need to do to get ready for our test?

detail of PDSA model

Do

In Step 6, your team will test the improvement theory on a small scale. Remember to document how the test unfolds.

Ask Yourself: Are we carrying out the test as we planned?

detail of PDSA model

Study

In Step 7, your team will use the data collected during the “Do” stage to study the results of your test. This process will include comparing results against your baseline data and discussing the team’s overall experience with the project.

Ask Yourself: Did the change we tested result in an improvement?

detail of PDSA model

Act

In Step 8, your team will decide what to do with the lessons learned through this PDSA cycle. The team can (1) adopt the change as a standard of practice, (2) test the change under different circumstances through a new PDSA cycle, or (3) abandon the change and try a new solution to the problem.

Ask Yourself: Should we keep the change and apply it more broadly, or do we need more information?

In Step 9, your team will work to build on the progress made by this PDSA cycle and make plans for additional cycles.

Ask Yourself: What did we accomplish in this PDSA cycle and how can we keep moving forward?


Frequently Asked Questions: Beginning CQI

Frequently Asked Questions: Beginning CQI

Q: My team finds ways to do things better all the time—isn’t that continuous quality improvement (CQI)?

A: Not necessarily. Although finding better ways to do things is a great quality in a team, true CQI is structured, ongoing, and measurable: “Quality Improvement is the use of a deliberate and defined improvement process, such as Plan-Do-Study-Act [PDSA], which is focused on activities that are responsive to community needs and improving population health. It refers to a continuous and ongoing effort to achieve measureable improvements in the efficiency, effectiveness, performance, accountability, outcomes, and other indicators of quality in services or processes which achieve equity and improve the health of the community.” [1] CQI refers to a formal process rather than the general efforts we all make to get better at our work. It should be used to improve processes and services in a way that can be measured through the use of data.

 

Q: How do I get program leadership and team members to buy in to CQI?

A: When trying to build buy-in and commitment to a new CQI process, leadership and staff may wonder, “What’s in it for us?” Take this into consideration and remind everyone that CQI efforts strengthen the connection between their everyday work and measurable results. CQI can increase efficiency and effectiveness, reduce costs and redundancies, improve productivity, streamline processes, increase customer satisfaction and employee morale, and improve outcomes.

 

Q: How can we squeeze CQI into our already busy schedules?

A: This is a common concern, but CQI doesn’t have to take up a lot of resources. First, embrace a team approach by splitting up the work among all teammates. This approach reduces the burden on any one person while also exposing more team members to CQI and building your organization’s overall capacity. Next, rather than creating extra meetings for your team, piggyback your CQI work onto existing group or committee meetings making the most out of times that are already blocked off. Set realistic goals that focus on change that is within your team’s control and appropriate given the time and resources available. Lastly, remember that CQI projects aim to improve processes by streamlining and finding efficient ways to meet your goals. Results may not be immediate, but a commitment to CQI can translate to meaningful improvements over time.

 

Q: CQI is new for us. Where do we start?

A: First of all, congratulations on beginning to explore the benefits of CQI! CQI doesn’t have to be complicated, and you have a lot of resources at your disposal. To start, you’ll want to gather basic information about CQI and approach your organization’s leadership for approval. We recommend Embracing Quality in Public Health: A Practitioner’s Quality Improvement Guide. Read about how other organizations have implemented CQI on the Public Health Quality Improvement Exchange. Look to your data (including numerical data like client records, results from customer or employee satisfaction surveys, or general staff feedback) to identify an area with room for improvement. At that point, you’ve already started Step One (“Getting Started”) of the Plan-Do-Study-Act cycle. Keep the momentum going by assembling your team and continuing through the PDSA stages.

 

Q: What do you do if your CQI results don’t meet your expectations?

A: This can happen and is something that most CQI teams will experience at some point. Carefully selecting the measures you will use to track improvement during the “Plan” stage is a critical step towards understanding whether your project resulted in the improvement you are seeking. If you are not seeing the trend in your data that you were expecting, it could be because you haven’t tested your improvement long enough or under the right parameters (e.g., the data might change at different times of the year due to reasons outside of your control, the data are only reflective of certain employees/customers, etc.). You may need to extend your timeline or test your improvement under different circumstances. If these steps don’t help, consider tackling a different root cause or testing a new solution that will get you closer to your goal. It can take multiple PDSA cycles to see improvement. If you are trying to address a large process improvement, consider breaking it down into several smaller PDSA cycles.


FOOTNOTE

[1] Bialek, R., Bietsch, L. M., Cofsky, A., Corso, A., Corso, L., Moran, J., Riley, W., & Russo, P. (2009). Proceedings from the Accreditation Coalition Workgroup: Quality Improvement in Public Health. Retrieved from http://www.rwjf.org/publichealth/product.jsp?id=48851


Building CQI Capacity To Improve Tribal Home Visiting

Building CQI Capacity To Improve Tribal Home Visiting

James Bell Associates has partnered with the Michigan Public Health Institute and Zero to Three to support and provide expertise to Tribal Home Visiting Program grantees on continuous quality improvement (CQI).

To begin their CQI work, grantees attend a 3-day intensive regional workshop for a hands-on introduction to CQI and the Plan-Do-Study-Act (PDSA) approach. Grantee teams learn about CQI concepts and tools and begin a CQI project. They continue the project when they return to their communities.

Grantees select a topic and measures for their CQI project based on their program goals. Common topics include—

  • Family engagement (e.g., family referrals, enrollment, retention, completed visits, participation in group meetings)
  • Screenings (e.g., intimate partner violence, maternal depression, substance abuse, child developmental delay, basic family needs)
  • Referrals (i.e., whether positive screenings result in accurate and timely referrals for appropriate services)

TEI follows up the regional workshops with quarterly Community of Learning Webinars that help grantees share their CQI work and learn about effective improvement approaches from peers and experts in the field.

Why Is CQI Helpful?

Built on values such as consensus, team building, and supportive learning, CQI develops program-level solutions to program-level problems. Grantees work together in teams to test changes aimed at streamlining processes, reducing redundancy, enhancing services, and improving outcomes for program participants.



Tribal Home Visiting Program Report to Congress

Tribal Home Visiting Program Report to Congress

Cover of the Tribal, Maternal, Infant and Early Childhood Home Visiting: A Report to CongressThe Tribal Home Visiting Program submitted its first report to Congress in February 2016. The report outlines the program’s approach, implementation, and findings since its initial funding in 2010 as part of the Affordable Care Act.

Twenty-five tribal organizations have received grants to date. The report describes the grantees, the context in which they work, and the families they serve. It also discusses the important ways in which the grant has built their capacity to develop, implement, and evaluate home visiting.

The program’s accomplishments include expanding home visiting services to high-need families across diverse communities, demonstrating improvement in key child and family outcomes, and building strong foundations for early childhood systems of care. The report details these strides and shows how tribally and community-driven programming and decision making promote innovation. It illustrates the ability of communities to collect and use data, demonstrate performance improvement, and implement evidence-based practices.

Downloads


Evaluation in Practice: Developing an Evaluation Plan With Scientific and Cultural Rigor

EVALUATION IN PRACTICE

Developing an Evaluation Plan With Scientific and Cultural Rigor

An interview with Native American Professional Parent Resources, Inc. (NAPPR), staff

Staff photo of Native American Professional Parent Resources, Inc. (NAPPR)

What is your evaluation question?

Do Native families participating in tribal home visiting that receive a culturally enhanced version of Parents As Teachers (PAT) (parent-child activities and family group connections) demonstrate increases in cultural self-efficacy, cultural interest, and cultural connectedness compared with Native families that receive standard (non-culturally enhanced) PAT through Early Head Start?

How did you balance cultural and scientific rigor when developing your evaluation plan?

First, it took time to develop internal evaluation capacity and mutual understanding among university evaluators and NAPPR staff. It was important for us to allow ample time to form trusting relationships and build shared ownership and investment in the research process. It also took time for the program to stabilize so that outcomes could be evaluated effectively.

We had to find the right study focus and research question. Once we determined that our outcome of interest would be “‘cultural connectedness,'” we had to decide how we were going to measure such a complicated construct. We chose to develop our own measure, consulting with and drawing on the work of other researchers. In the process of developing cultural enhancements, we had to navigate tribal governance systems: Who has authority to call a culturally enhanced activity “Pueblo”? How can someone get that authorization? Consulting with our home visiting model developer about enhancing the curriculum to include culturally-tailored home visit and group activities took time as well.

Throughout the process, we consulted with the our program’s community advisory board, parent advisory group, and staff. There was an ongoing feedback loop with these groups. We wanted their input and consultation at every stage of development, so the study became a regular item on meeting agendas.

How did your commitment to balancing cultural and scientific rigor influence decisions you made about the evaluation?

Balancing cultural and scientific rigor was a study-long process. We questioned what we could do to be more culturally responsive each step of the way. For example, because we serve a population that is tribally diverse, we decided against the idea of developing tribal-specific cultural activities. Instead, we developed intertribal activities that would appeal to participants from different tribes with prompts for families to share their own tribal values and traditions. By designing our intervention to be more intertribal, we decided that our home visitors were not going to be teachers but facilitators for cultural activities. This was important for our evaluation, because it meant the intervention would vary somewhat from family to family. Having a tribally diverse population also meant the definition of “cultural connectedness” could vary among participants. We worked hard to develop survey language relevant to participants from a range of tribes. We also built focus groups into our evaluation design, in addition to surveys, to capture the diverse ways participants perceive and experience cultural connectedness.

How did TEI help?

TEI helped us understand federal expectations and supported us in finding the right evaluation focus for our program and outlining a preliminary evaluation plan. TEI also supported us in achieving a good balance of cultural and scientific rigor, often by asking questions that prompted us to rethink proposed approaches and reach for greater rigor, but also by acknowledging our progress and successes along the way.


Demystifying Peer Review

Demystifying Peer Review

Peer review is a process for ensuring that the research published in journals is of high quality. This brief describes the benefits and process of peer review. It also explains how to write an article for a peer-reviewed journal.

The brief was developed to support Tribal Home Visiting Program grantees, including managers, evaluators, staff, and partners. It also may be useful for other professionals in home visiting and other health and human services fields. Whether you are new to peer review, already have some knowledge of it, or need a tool to help you explain it to others, this brief can help.

After reading this brief, you will be able to —

  • Identify the benefits of peer review
  • Describe the peer review process
  • Consider writing an article for a peer-reviewed journal


Presenting at a Conference

Presenting at a Conference

Two women talking in front of a conference poster.

When program staff and evaluators give presentations at conferences, they share knowledge, highlight their program’s achievements and their community’s strengths, and reinforce their skills and confidence. Collaborating on presentations can make the program team stronger.

This brief demonstrates the value of presenting at conferences and explains how to do it successfully. It was developed to support Tribal Home Visiting Program grantees, but it also may  be useful for other professionals in home visiting and other health and human services fields.

The brief will help program staff —

  • Understand the benefits of presenting at a conference
  • Identify and assess topics for a presentation
  • Find the best conference for a presentation
  • Plan a presentation, from getting approvals to arriving on site
  • Present effectively and confidently
  • Document their efforts and continuously strengthen their presentation messages, formats, and skills