Good News

TEI Publishes Tribal Home Visiting Grantee Evaluation Plan Profiles on OPRE Website

The Tribal Evaluation Institute (TEI) created a compilation of Tribal Home Visiting Program evaluation plan profiles that introduce and describe the grantees’ evaluation studies. The profiles are designed for evaluators, program implementers, and federal staff interested in program impact in complex community contexts and may be most useful for individuals thinking about evaluating tribal home visiting and/or early education initiatives. They can be used as resources for the development and implementation of future evaluations in tribal communities, as well as to inform evaluation-related policies and grant requirements for tribal funding recipients. Each grantee profile presents a description of the program, a summary of their evaluation plan, and information about their evaluation team.

You can view the published report on the OPRE website.


TEI Contributes to Special Issue of the Infant Mental Health Journal

Staff from the Tribal Evaluation Institute (TEI) project worked closely with Tribal Home Visiting grantees, ACF, and the Tribal Early Childhood Research Center (TRC) to contribute to five articles in the May/June 2018 volume of the Infant Mental Health Journal titled, “New Directions in Infant and Early Childhood Home Visiting: Lessons Learned from the Tribal Maternal, Infant, and Early Childhood Home Visiting Program.” The published articles include:

  • Building evaluation partnerships with tribal communities for home visiting (Ayoub, et al.)
  • Methodological considerations for home-visiting research in tribal communities (Kilburn, et al.)
  • Performance measurement in Tribal Home Visiting: Challenges and opportunities (Morales, et al.)
  • Measurement issues in home visiting research within tribal communities: Challenges and strategies (Rumbaugh Whitesell, et al.)
  • Approaches to the evaluation of cultural adaptations of home visiting in tribal communities (Meyer, et al.)

This journal issue represents the first time tribal home visiting efforts were the focus of a special issue.


Introducing the Data System Improvement Toolkit!

The Tribal Evaluation Institute (TEI) is pleased to announce the release of our Data System Improvement Toolkit! This resource is designed for tribal programs planning to develop or improve their data systems. Program and evaluation staff who have experience with and oversee changes to their existing data systems will find this toolkit informative and helpful.

The Data System Improvement Toolkit includes five modules.

Module 1: Choosing a System and Working with a Vendor or Developer

Module 2: Documenting and Improving Data System Processes

Module 3: Protecting Data Ownership and Privacy

Module 4: Displaying and Reporting Data

Module 5: Optimizing Your Current Data System
The full toolkit is available as a PDF file or access individual modules to download modifiable versions of the tools. Some readers will find that one module or even one tool can address their needs. Others may find it helpful to access each module in the order they are presented. For many, the right approach lies somewhere in between.

The TEI team is available to provide technical assistance for this toolkit and other data system needs. Please contact Erin Geary at 218-464-1260 or geary@jbassoc.com.


Tribal Evaluation Institute Presents at National Conferences

Tribal Evaluation Institute staff members presented at three national conferences in November and December, 2017. These presentations highlighted work that TEI is doing to support local evaluation capacity and improve data systems.

 

Zero to Three Annual ConferencePhoto of presenters at the Zero to Three Annual Conference, 2017

Brandie Buckless, M.P.H. attended the 2017 Zero to Three Annual Conference, November 29 - December 1 in San Deigo, CA.. She presented collaboratively with Dr. Catherine C. Ayoub from the Brazelton Touchpoints Center and Pamela Gutman from the Cherokee Nation on a session titled Tribal Home Visiting Evaluations: Moving the Field Forward. The session highlighted Tribal Home Visiting grantees' rigorous evaluations to examine cultural adaptations and enhancements made to evidence-based home visiting models. Presenters discussed preliminary evaluation findings and the use of innovative methods.

 

American Evaluation Association Annual Conference

TEI Deputy Director, Julie Morales, Ph.D. participated in a panel presentation on evaluation capacity building at the American Evaluation Association Annual Conference in Washington, D.C. in November 2017. Dr. Morales provided an overview of TEI’s approach to working with tribal grantees to build evaluation capacity over 7 years. Dr. Morales shared examples of TEI’s work including the Data Collection Toolkit.

 

Administration for Children and Families Native American Grantee Meeting

Erin Geary, Ph.D. participated in a session at the 2017 Administration for Children and Families Native American Grantee Meeting, November, 2017 in Washington, D.C.. The panel presentation was titled Community-Engaged Research and Evaluation Principles and Values in Tribal Home Visiting. The panel included presentations by Aleta Meyer, Ph.D., from the Office for Planning, Research, and Evaluation and Moushumi Beltangady, M.P.H., M.S.W., from the Office of the Deputy Assistant Secretary for Early Childhood Development at the Administration for Children and Families. Dr. Geary’s presentation focused on data system focused technical assistance available to tribal home visiting grantees.


Tribal Home Visiting Program Report to Congress

Tribal Home Visiting Program Report to Congress

Cover of the Tribal, Maternal, Infant and Early Childhood Home Visiting: A Report to CongressThe Tribal Home Visiting Program submitted its first report to Congress in February 2016. The report outlines the program’s approach, implementation, and findings since its initial funding in 2010 as part of the Affordable Care Act.

Twenty-five tribal organizations have received grants to date. The report describes the grantees, the context in which they work, and the families they serve. It also discusses the important ways in which the grant has built their capacity to develop, implement, and evaluate home visiting.

The program’s accomplishments include expanding home visiting services to high-need families across diverse communities, demonstrating improvement in key child and family outcomes, and building strong foundations for early childhood systems of care. The report details these strides and shows how tribally and community-driven programming and decision making promote innovation. It illustrates the ability of communities to collect and use data, demonstrate performance improvement, and implement evidence-based practices.

Downloads


Evaluation in Practice: Developing an Evaluation Plan With Scientific and Cultural Rigor

EVALUATION IN PRACTICE

Developing an Evaluation Plan With Scientific and Cultural Rigor

An interview with Native American Professional Parent Resources, Inc. (NAPPR), staff

Staff photo of Native American Professional Parent Resources, Inc. (NAPPR)

What is your evaluation question?

Do Native families participating in tribal home visiting that receive a culturally enhanced version of Parents As Teachers (PAT) (parent-child activities and family group connections) demonstrate increases in cultural self-efficacy, cultural interest, and cultural connectedness compared with Native families that receive standard (non-culturally enhanced) PAT through Early Head Start?

How did you balance cultural and scientific rigor when developing your evaluation plan?

First, it took time to develop internal evaluation capacity and mutual understanding among university evaluators and NAPPR staff. It was important for us to allow ample time to form trusting relationships and build shared ownership and investment in the research process. It also took time for the program to stabilize so that outcomes could be evaluated effectively.

We had to find the right study focus and research question. Once we determined that our outcome of interest would be “‘cultural connectedness,'” we had to decide how we were going to measure such a complicated construct. We chose to develop our own measure, consulting with and drawing on the work of other researchers. In the process of developing cultural enhancements, we had to navigate tribal governance systems: Who has authority to call a culturally enhanced activity “Pueblo”? How can someone get that authorization? Consulting with our home visiting model developer about enhancing the curriculum to include culturally-tailored home visit and group activities took time as well.

Throughout the process, we consulted with the our program’s community advisory board, parent advisory group, and staff. There was an ongoing feedback loop with these groups. We wanted their input and consultation at every stage of development, so the study became a regular item on meeting agendas.

How did your commitment to balancing cultural and scientific rigor influence decisions you made about the evaluation?

Balancing cultural and scientific rigor was a study-long process. We questioned what we could do to be more culturally responsive each step of the way. For example, because we serve a population that is tribally diverse, we decided against the idea of developing tribal-specific cultural activities. Instead, we developed intertribal activities that would appeal to participants from different tribes with prompts for families to share their own tribal values and traditions. By designing our intervention to be more intertribal, we decided that our home visitors were not going to be teachers but facilitators for cultural activities. This was important for our evaluation, because it meant the intervention would vary somewhat from family to family. Having a tribally diverse population also meant the definition of “cultural connectedness” could vary among participants. We worked hard to develop survey language relevant to participants from a range of tribes. We also built focus groups into our evaluation design, in addition to surveys, to capture the diverse ways participants perceive and experience cultural connectedness.

How did TEI help?

TEI helped us understand federal expectations and supported us in finding the right evaluation focus for our program and outlining a preliminary evaluation plan. TEI also supported us in achieving a good balance of cultural and scientific rigor, often by asking questions that prompted us to rethink proposed approaches and reach for greater rigor, but also by acknowledging our progress and successes along the way.


TEI Data Collection Toolkit

Data Collection in the Home: A TEI Toolkit

Print Friendly, PDF & Email

The data collection toolkit was developed to support data collection with AIAN families in their homes. Guided by years of work providing technical assistance to Tribal Home Visiting Program grantees, the toolkit addresses common grantee needs and challenges. Although it was designed for grantees—including program managers, evaluators, home visitors, and other staff—it also may be useful for early childhood programs and others who serve AIAN communities. The data collection toolkit supports culturally rigorous data collection.

The toolkit was designed to help programs—

  • Understand the value of data collection
  • Prepare for data collection
  • Collect high-quality data
  • Use tools to develop data collection processes, collect data, and implement quality assurance

Why Data Collection Is Important

Good decisions are driven by good data—information that is consistent, accurate, and complete. Quality data help programs tell stories about participating families, services, and outcomes that they can rely on to inform decision making.

Data collection has never been more important. Programs need data to apply for increasingly competitive funding opportunities, meet ambitious grant reporting requirements, and address participants’ needs with evidence-based strategies. Tribal Home Visiting Program grantees are required to collect data for continuous quality improvement, performance measurement, and program evaluation.

How To Navigate and Use the Toolkit

Introduction to the Data Collection Toolkit

The purpose of the toolkit, intended audiences and how to use it. Download Introduction in Microsoft Word format (.docx, 872kb).

Module 1: Understanding the Value of Data Collection

Training staff on the basics of data and how they can collect and use data. Download Module 1 in Microsoft Word format (.docx, 919kb).

Module 2: Preparing for Data Collection

Planning and building a foundation to collect quality data. Download Module 2 in Microsoft Word format (.docx, 919kb) and additional resources used in Module 2: Activity 2.2 Jeopardy Game (PowerPoint) and Tool 2.4 Data Collection Scheduler (Excel).

Module 3: Collecting High-Quality Data

Supervising data collection and implementing quality assurance. Download Module 3 in Microsoft Word format (.docx, 919kb) and an additional resource used in Module 3: Tool 3.11 Inter-Rater Agreement (Excel).

Toolkit Modules Representing Stages of Data Collection

TEI Toolkit Modules graphic

Intended Audiences


Program managers


Program managers may deal with data collection from planning and oversight to data entry and analysis. They typically make decisions and ensure that staff understand their role in data collection and are trained and supported. Open communication between program managers and staff is crucial for troubleshooting challenges. Program managers are often asked to present data to stakeholders and funders, so they must have a solid understanding of why data collection is important and how it works.


Data coordinators


Data coordinators (also called data managers) play a critical role in collecting, entering, managing, and reporting data. Data coordinators help home visitors keep track of which forms need to be filled out and when. Having a data coordinator to focus on data-related tasks maximizes the time home visitors and program managers can spend serving families.


Evaluators


Like program managers, evaluators ensure that staff appropriately use, interpret, and store data. They develop and implement guidelines for administering and interpreting evaluation instruments. Examples include writing data collection protocols, establishing consent processes, identifying and reviewing instruments, and selecting data systems. Evaluators may support data entry and analysis, data quality reviews, and reporting. They also help promote collaborative community-based evaluation practices.


Home visitors


Home visitors are the faces of the home visiting program in the community, and they are typically responsible for collecting data from program participants. Home visitors help ensure that the program collects high-quality data in a way that is comfortable for the families the program serves. They are often tasked with explaining data collection to families, administering data collection instruments, entering data into databases, and communicating assessment results to the families served by the program.