TEI

Evaluation Capacity Readiness Assessment Checklist

Tribal MIECHV Evaluation Capacity Readiness Assessment Checklist (2017)

The Evaluation Capacity Readiness Assessment Checklist (ECR) was developed to help Tribal MIECHV grantees prepare to evaluate their home visiting services. The ECR is a self-assessment tool focused on examining organizational and team readiness to engage in evaluation activities. The assessment process can help grantees determine the areas of their program and organization that may need to be strengthened to be fully prepared to plan, launch, and implement an evaluation. The ECR should be conducted before evaluation questions are developed, as the results can inform what is feasible and appropriate for the program and organization. While the ECR was developed for Tribal MIECHV grantees, it is applicable to any program seeking to evaluate their services and activities.

Print Friendly, PDF & Email

Download the ECR Tool

ECR Tool in PDF format

The purpose of the toolkit, intended audiences and how to use it. Download ECR Tool in PDF format (.docx, 377kb).

ECR Tool in Microsoft Word format

The purpose of the toolkit, intended audiences and how to use it. Download ECR Tool in Microsoft Word format (.docx, 377kb).


Data System Improvement Toolkit - Module 5

Number 5

Module 5: Optimizing Your Current Data System

Strengthen your existing data system by learning to conduct assessments, identify enhancements, and organize your overall improvement project. Tools marked with an * below can be modified.

MODULE CONTENT

Module Introduction

This text shares two steps for customizing a data system and provides in-depth descriptions of tools found within the module

5.1 Guide to Data System Self-Assessment

This guide describes data system assessment, when it should take place, and what core components to examine. Read before accessing tool 5.2. (Guide)

5.2 Data System Assessment Checklist

Use this checklist to facilitate a conversation about your data system’s requirements, strengths, and areas for improvement. (Checklist/assessment)

5.3 Data System Improvement Charter

This sample charter describes objectives, goals, timelines, and benefits of a data system improvement project. (Template)


Data System Improvement Toolkit - Module 4

Number 4

Module 4: Displaying and Reporting Data

Access tools for creating and refining data dashboards to monitor services and prepare reports for funders. Tools marked with an * below can be modified.

MODULE CONTENT

Module Introduction

This text provides basic information about data dashboards and describes tools found within the module.

4.1 Data Reporting Map*

Use this tool to determine where data points are pulled from in existing reports and how calculations are made. Programs may find it useful to enlist addition support from a technical assistance provider or other consultant. (Template)

4.2 Understanding System Integration

This guide describes different types of system integration and provides an overview of the planning process. (Guide)

4.3: Guide to Data Dashboards

This guide serves as an introduction to tool 4.4. It lists common dashboard elements and considerations for customizing dashboard data. (Guide)

4.4 Example of a Data Dashboard

This sample dashboard presents a “real-world” example of concepts outlined in tool 4.3. (Example)


Data System Improvement Toolkit - Module 3

Number 3

Module 3: Protecting Data Ownership and Privacy

Get up to speed on managing and protecting data—and how to address these issues with systems vendors and developers. Tools marked with an * below can be modified.

MODULE CONTENT

Module Introduction

This text details two steps for owning and protecting client information and lists tools found within the module.

3.1 Understanding Data Ownership

This guide describes core elements of data ownership, including the use of ownership agreements and governance planning. Use with tool 3.2 (Guide)

3.2 Example of Data Ownership Contract Language*

Structured around the core elements outlined in tool 3.1, this document provides sample contract language for data ownership agreements. (Example)

3.3 Guide to Data Privacy and Confidentiality

Read this guide to better understand what data to protect, what regulations to comply with, and how to work with system vendors to ensure privacy. (Guide)

3.4 Considerations for Cloud Versus On-Premise Software and Storage

This tool explains the differences between cloud-based and on-the-premises storage and software options in areas such as cost, system function, ownership, and security. (Guide)


Data System Improvement Toolkit - Module 2

Number 2

Module 2: Documenting and Improving Data System Processes

Consistency is key when improving data systems and documenting related processes. Learn how to streamline data entry, quality, and storage. Tools marked with an * below can be modified.

MODULE CONTENT

Module Introduction

This text outlines three steps for strengthening data system processes and describes tools found within the module.

2.1 Guide to Data Mapping

This short guide includes a step-by-step overview of data mapping. Review it, along with tool 2.2, before working with a vendor to map your data system. (Guide)

2.2 Data Map Template*

Once you’ve read tool 2.1, use this customizable template to create a visual map of the data in your system. (Template)

2.3 Guide to Data Cleaning

This document provides a brief overview of data cleaning, including specific tasks for program staff, supervisors, and managers to complete before and during data reporting. (Guide)

2.4 Example of a Case File Checklist*

Supervisors can use this customizable checklist when reviewing case files to see if assessments were performed completely and on time. Use with tool 2.5. (Example and Template)

2.5 Example of a Missing Data Report*

This customizable template features a sample missing data report to help programs summarize results from their case file reviews. (Example and Template)

2.6 Data System Business Process Maps*

Process maps help users better document data collection and entry related to intake, service provision, and reporting. They can be customized with the help of a technical assistance provider. (Example and Template)

2.7 Best Practices When Converting to a Paperless Data System

This brief guide presents best practices and common challenges when moving from a paper-based data collection and entry system to a paperless system that uses laptops or tablets. (Guide)


Data System Improvement Toolkit - Module 1

Number 1

Module 1: Choosing a System and Working with a Vendor or Developer

Is it better to use a commercial off-the-shelf (COTS) product or develop a new, customized one? Make an informed decision and translate it into action. Tools marked with an * below can be modified.

MODULE CONTENT

Module Introduction

This text defines terms commonly used when selecting a data system and provides in-depth descriptions of tools found within the module.

1.1 Decision Guide: Is a COTS System Right for You?*

This table of questions can help you determine whether a COTS or custom data system would better meet your program requirements. (Guide and assessment)

1.2 Scan of COTS Systems*

This tool provides a summary of commonly used COTS products, including those used by tribal and state MIECHV grantees. (Summary)

1.3 Request for Proposals – Timeline Guide

This template identifies the major activities associated with developing a request for proposals. Visit tool 1.5 for information on vendor and developer activities. (Template)

1.4 Request for Proposals – Timeline Template*

Once you’ve identified your timeline using tool 1.3, enter your dates into this customizable template to create a personalized timeline chart. You can also add columns to note staff responsibilities, additional information, etc. (Template)

1.5 Request for Proposals – Template*

This template helps teams looking to hire a system vendor or developer. Complete the template by adding dates and information from tool 1.4. (Template)

1.6 Example of a Data System Software License Agreement

This tool walks you through the key elements of software user agreements, which often contain important, non-negotiable language about vendor expectations. (Example)


Data Systems Improvement Toolkit

Data System Improvement Toolkit

The Data System Improvement Toolkit is designed to help tribal program grantees develop or improve data systems. Program and evaluation staff who have experience with and oversee changes to their existing data system will find this toolkit informative. If you are interested in using the toolkit but do not have experience with data systems, please contact your technical assistance provider assigned through your federal grant.

How to Navigate the Toolkit

The Data System Improvement Toolkit contains five modules that walk users through the steps of strengthening their data system. Each module includes the following:

  • Introduction, including goals and terminology
  • Overview of the tools
  • Copies of the tools, including modifiable versions when applicable

Click on a module below to access its content. You can also download the full toolkit in PDF, including a more in-depth introduction.

How to Use the Toolkit

The five modules in this toolkit represent steps in a cycle. However, data system improvement is ongoing with no defined beginning or end. The following questions can help programs choose the right starting point for their needs:

  1. Does your system currently meet your requirements or do you need something new?
  2. What challenges most impact your ability to serve families?
  3. What improvements do you need to focus on most, in keeping with available time and resources?

Some programs will find that using one module or even one tool can address their needs. Others may find it helpful to access each module in the order they’re presented. For many, the right approach lies somewhere in between.


Data Systems Cycle graphic

MODULE OVERVIEW

Module 1: Choosing a System and Working with a Vendor or Developer

Is it better to use a commercial off-the-shelf (COTS) product or develop a new, customized one? Make an informed decision and translate it into action.

Module 2: Documenting and Improving Data System Processes

Consistency is key when improving data systems and documenting related processes. Learn how to streamline data entry, quality, and storage.

Module 3: Protecting Data Ownership and Privacy

Get up to speed on managing and protecting data—and how to address these issues with systems vendors and developers.

Module 4: Displaying and Reporting Data

Access tools for creating and refining data dashboards to monitor services and prepare reports for funders.

Module 5: Optimizing Your Current Data System

Strengthen your existing data system by learning to conduct assessments, identify enhancements, and organize your overall improvement project.


Summary of Performance Measurement Plans

Summary of Performance Measurement Plans

TEI works with Tribal Home Visiting Program grantees to help them develop plans to measure the performance of their programs. This document summarizes 25 grantee plans (for grantees funded between 2010 and 2016) to measure improvements in 6 benchmark areas:

  1. Maternal, newborn, and child health
  2. Child injuries; child abuse, neglect, or maltreatment; and emergency department visits
  3. School readiness and achievement
  4. Crime or domestic violence
  5. Family economic self-sufficiency
  6. Coordination and referrals for other community resources and supports

Each area looks at specific concepts. For example, what percentage of recommended prenatal care visits did mothers receive? How many children visited the emergency department? What percentage of parents were screened for depression? What was participants’ health insurance status?

The summary reviews the types of measures the grantees selected and the data sources, populations, tools or indicators, and time periods for data collection. It also describes the types of comparisons used to demonstrate improvement, direction of improvement needed to determine success, units of analysis, and types of scoring.

Grantees funded between 2010 and 2016 had the flexibility to develop performance measures that were meaningful to them and met their needs. This flexibility was a plus for the programs and communities, but the variety of measures presented some challenges. It was time consuming and stressful picking the best measure. It was also difficult to combine and compare results across grantees. The summary identifies common themes and similarities among grantees’ performance measures that made it easier to draw broad conclusions. For example, most grantees chose to measure duration of breastfeeding, making it possible to compare their results.


What Is Continuous Quality Improvement?

What Is Continuous Quality Improvement?

Continuous quality improvement (CQI) is an ongoing process for achieving measurable improvement. It is a proven technique for increasing efficiency and productivity, increasing participant and stakeholder satisfaction, building employee morale, and improving outcomes. CQI connects everyday work with results.

This presentation defines quality improvement, describes what it can do for you, explains the difference between quality assurance and quality improvement, shares guiding principles and questions, and introduces the Plan-Do-Study-Act cycle.

TEI’s What Is CQI – (May 2016) video from James Bell Associates on Vimeo.


Engaging Communities in the Development of Performance Measurement Plans

Engaging Communities in the Development of Performance Measurement Plans

Performance measures, also called benchmarks, are indicators that help track program performance and improvement. Tribal Home Visiting Program grantees engage their communities in the development of performance measures that provide useful data for program management, continuous quality improvement, and ongoing assessment of family needs.

ICON-dialog-2b“The Community Council assisted in framing questions for our demographic and visitation forms, both of which collect performance measurement data. We went through each question with the council, discussing the order of questions, purpose, wording, rephrasing, and what data the council wanted us to collect.”

— Tribal home visiting grantee

Engagement Strategy 1: Incorporating information from a community needs assessment

A community needs assessment is a systematic way to determine the current needs of the local community. Through a needs assessment, communities gather information using various methods (e.g. record review, focus groups, interviews) to drive program priorities and performance measurement planning. The feedback loops created during the assessment are valuable for gathering community input throughout the performance measurement planning process.

Engagement Strategy 2: Using community advisory groups

Most grantees form and rely on community advisory groups during the development of the performance measurement plans. Involvement of the advisory group varies across grantees. Some grantees engage with the group throughout plan development, while others use the group in the end stages to review and approve plans.

Engagement Strategy 3: Engaging community members in the review of potential measures and measurement tools

Given the lack of measures validated with AIAN populations, community input is often the only way to know how program participants will interpret the terms and concepts in measurement tools. It is best to engage community members in a multistep process for choosing, refining, and obtaining approval for measures and tools.

Engagement Strategy 4: Involving program staff and other stakeholders

In addition to formally engaging community members, grantees can draw on community voices from within their program and from partner agencies. This strategy provides an opportunity to gather staff perspectives and to discuss data collection with other programs that serve families in the community.