Program Evaluation 3 – Literature Review

Having selected a program to evaluate and defined the program’s rationale, we turn to the literature review. A well done literature review will identify issues to research and generate new knowledge and insights that should lead to a cutting-edge program. Here’s how to do one well.

How a Literature Review Helps

A literature review will:

• Broaden your thinking by providing context for the issue. It’s possible to be so consumed with the immediate work at hand that you miss the big picture. A lit review will correct that. An inner-city mission might be founded to feed the hungry. As they read about hunger they will find that it is very much related to poverty and poverty is related to several issues, including mental health and joblessness. So the mission might add counselling and job skills training to its services. But causes related to individuals are just part of the problem because there are systemic reasons why poverty exists. So the inner-city mission may engage in advocacy work, seeking legislative change or systemic change that will help minimize poverty. A literature review will help you think BIG BROAD thoughts!
• Take you deeper into the nuances of the issue. It will help you think DEEP thoughts. While writing my dissertation about church-agency relations, I started with the understanding that the divisive issue between them was one of control. Most of the authors framed their discussions this way, and yet as I did the literature review and dug below the surface of their arguments, I realized control wasn’t the issue – responsible relationship was. The solution to the problem could not be resolved if the debate centred on control, but it is easily resolved once we address the underlying goal that was shared by every one of the authors. This is the subject of my book The Church At Work: A manual for church-agency relationships that will be coming out by the end of the year. Exploring the breadth and depth of your mission will stimulate all sorts of creative ideas.
  
• Reveal the points of controversy and the resulting key choices you need to make. Let’s say you are a Bible study ministry producing educational materials. A literature review will show that adults and children learn differently and that there are various theories of learning that compete with each other. There are methods of studying the Bible that compete with each other too. There is even controversy over which version of the Bible is the best one to use for study. There are many choices to be made in each of these topic areas and by understanding the points of controversy or divergence, you can make an intelligent decision for what your ministry will do and you’ll know why it is the best choice.
• Keep you up to date in your field. Missiology, for example, has changed enormously in the last 50 years, and even the last 10 years. If you are a sending agency and are still using a missions model developed in the 1950s or 1960s, you are badly out of date. Missiologists have changed their thinking on relationships between sending and receiving countries and created a new paradigm for global mission. Not only has the theory changed, but so have the practices. A lit review will help you be a leading-edge ministry.

What to Do a Literature Review On

Review your theory of change and logic model to find researchable topics. The topics might not be explicitly listed, but look at each box and ask, “What topics are related to this box?” Based on my previous post about the program review of our annual conference, some possible researchable topics for our program review include:

• conferences: trends, promotion, logistics, why people attend, business models etc.
• adult learning theory and converting learning into doing
• networking

How to Find the Literature

• Books often have the most thorough treatments of a topic. Look especially for recent books based on primary research. For help in selecting books, read this post. Don’t forget to go to the library to see what you can borrow before buying a bunch of books.
• Online websites, blogs and journals are where you will likely find the most up-to-date material. If you are referred to journals that only have excerpts, or if you have to pay for the information, check if your your local university, college or seminary subscribes to the journal. If not, you can likely do the search from within the library and see the results because they have already paid for access to databases of journals and you can read them or print them out at the library.
• Ask people to recommend sources for you. If they are in the same field as you, they can usually recommend either a title or an author.
• You can stop looking for additional literature when you find that all of the citations and references are to articles and books you’ve already seen, or when you find the material has become repetitious.

How to Analyze the Literature

First of all, the purpose of this literature review is to design better programs and services. For a real-life excellent example of how to do the lit review, see Andy Harrington’s example which I highlight here. Since you are not doing an academic literature review, the only issue to resolve is whether or not the information is useful to your ministry. How you analyze the literature depends on whether you are reading about application and execution (to tweak a program) or about ideas and theories (to test the design of the program).

If you are looking for application and execution ideas, you simply make a list of the ones you find in the lit review that you want to consider. For example, a list of technologies that can enhance a conference experience. If you are researching at the theoretical or idea level, then you have some additional work to do to get below the surface of differing opinions:

• What are the premises on which the various opinions are based? Are they valid? Do they apply to your situation? Especially in the social and political spheres, you have to ask if the premises and conclusions are theologically acceptable.
• What are the authors’ goals and values? Can you reconcile their various views? Can some be discarded because they conflict with your goals and values? Sometimes their goals and values are explicit and sometimes not, in which case you have to try to infer them. In my dissertation research on church-agency relations I discovered that some authors were primarily concerned that individuals be able to use their gifts while others were more concerned about showing unity to the public. Some had a Christological focus while others had a Trinitarian focus. Some used very strong language heavy on the ‘control’ aspect yet when they gave examples of what the relationship should be, the examples were much more moderate in tone. I was able to reconcile the positions of authors who at first appeared to be diametrically opposed to each other. The result was a new model of church-agency relations that all of the authors should be able to live with. In fact, one of the authors I critiqued read my work and said he agreed with the end result even as he disagreed on how I got there!
• Finally, look at the recommended action steps or the model they propose. Have they made a convincing argument to support their recommendations? Do they make sense to you?

 The Outcomes of a Literature Review

• A literature review should make you aware of what the choices are for how your program is designed and delivered. Based on this, the program review would then assess if the current design would benefit from a change.
• It may determine questions you should ask as you do the program evaluation.
• It should help you identify hidden assumptions you have made, which you can then test to see if they are valid.
• It might suggest that some further research is needed in a particular area.
• It could provide a shopping list of enhancements to the current program or a list of ideas for new programs and services.

Conducting a literature review will help ensure that, as you progress through the program evaluation, you are well informed and up to date on the subject you are evaluating, and it will help you focus on the areas that need more scrutiny.