From the IPA Blog

Posting Hypotheses for an Impact Study of Compartamos

Mar 12/12 | From the blog
by Dean Karlan

We are nearing completion of a randomized trial of the impact of Compartamos, a for-profit microlender in Mexico, and their expansion in the Nogales area. We will be posting our hypothesis before we do the analysis, and encourage readers to do the same, for three reasons:

  1. Avoid data mining: Prespecifying hypotheses is a helpful way of addressing a data mining concern when multiple outcomes are measured. By prespecifying primary and secondary outcomes, it is transparent whether 5 out of 10 hypotheses came true or 5 out of 100. With proper statistics, 5 out of 10 is not a fluke, but 5 out of 100 is exactly what one would call a fluke and is a kin to finding no changes at all.  So prespecifying is helpful: when it comes time for publication, people reading the research know that any significant results found were not simply as a result of mining the data for the magic significant relationships. Development economists have been slow to do integrate this step into our research; folks have pointed this out in several posts around the web (see here and here, for example). A registry system is indeed underway, and we’re quite supportive of this.
     
  2. Comparing results to prior opinions: The goal of research is to produce knowledge. But we aren’t in a vacuum beforehand, and we of course have our opinions. Suppose we are measuring where something is on a scale of -10 to 10. We have a prior opinion that it is 7. We lack evidence on that, so we are uncertain, but we still have an opinion that it is 7. If the result comes out and is 7, of course the study should not change our opinion. If on the other hand the result comes out and is 4, or 0 or -3, we ought to update our opinion at least a bit, perhaps not all the way. In this spirit, we are also welcoming the posting of predictions, preferably signed, so that if you are right you can show the world “I told you so” and if not, well then maybe it’ll encourage you to shift your opinion a bit. If this works, maybe we’ll start a website called www.itoldyouso.com (although that url is taken by someone not using it). 
     
  3. Crowdsourcing:  We’d like to hear from people their thoughts on which of the outcome measures they think will be particularly important. A simple way of engaging with the microcredit community ex-ante, rather than ex-post, with a hope of creating a better dialogue than one typically sees on studies after the results are posted. Plus we’re interested to hear thoughts from people, that may shape our analysis (although pre-specifying the hypotheses we’ll post of course!).

You can look forward to more blog posts with our hypotheses on different subjects, coming out in the next few weeks.

The Study

IPA has partnered with Compartamos Banco in Mexico to evaluate the social and economic impact of Crédito Mujer, their principal village banking loan product. The product offers individual women access to credit from $1,500 to $27,000 Mexican pesos (1 US$ = 12.8 Mexican pesos). Compartamos employs a group process, with women organized in groups of ten to fifty to act as solidarity guarantors.

This study took advantage of Compartamos’ decision to open three new branches in northern Sonora, where it had not previously operated. Within the region, we created 250 clusters and randomly assigned 125 of them to receive direct promotion of Crédito Mujer, while the remainder served as the control and did not receive any direct promotion. To ensure differential take-up between treatment and control groups, Compartamos restricted loan access exclusively to treatment clusters; loan officers, coordinators, and branch managers were responsible for physically verifying the addresses of all potential clients before they could take out credit.

After Compartamos offered credit to women in northern Sonora treatment areas for at least 18 months, we launched a survey to measure the impact of Crédito Mujer, through a survey which asks respondents’ about income, well-being and businesses. We started fieldwork on surveys in November, and plan to complete interviews with 16,500 women by the end of March.

The Sample Population

The women in our sample range in age from 18-60 years old. At the time of our baseline survey in 2010, the average age was 41 (st dev: 15). Half of the sample was married, and the women had, on average, 1.1 children (st dev: 1.1).

Before Compartamos Banco began marketing, according to our baseline survey, 24.5% of the sample owned a business. Two-thirds of the sample reported that they were unlikely to take out credit in the next 6 months, and one-quarter said they were likely to take out credit. 18.9% reported having taken a loan in the previous year, with the majority of those loans (44%) coming from a bank or finance institution, and the rest coming from other sources, like moneylenders, relatives, store credit, or a friend. Compartamos Banco’s Crédito Mujer product introduced as a new opportunity for the majority of the sample.

The Survey Instrument

Here is the survey instrument:

There are 22 main sections:

I. Personal characteristics
II. Household population
III. Health
IV. Household characteristics
V. Children
VI. Migration inside and outside Mexico
VII. Household consumption and assets
VIII. Savings
IX. Household assets
X. Business information
XI. Business experience
XII. Overall satisfaction
XIII. Income
XIV. Social networks
XV. Community and political engagement
XVI. Decision-making
XVII. Locus of control
XVIII. Unexpected expenses and events
XIX. Sources of credit
XX. Bank account
XXI. Credits
XXII. Mood in the last week

Feedback from the Microcredit Community

Please email or post one or two of the following:

  1. Which outcome measures do you consider to be the most important, i.e., the primary ones on which the overall success (or failure) of the program should be judged?
  2. Do you have any specific predictions on any particular outcomes? Or, put it the other way, are there any outcome measures for which if the results are outside of a certain range you will change your opinion?



 

Comments

The books should be available

The books should be available online at Topatoco in the near future! We'll keep you posted.

mark

its a great Survey Done !

Hi Stephanie -- We responded

Hi Stephanie -- We responded to your comment here: http://www.poverty-action.org/node/5135. Let us know what you think!

7 Hypotheses...6 too many?

This is interesting, thanks for opening it up. So my hypotheses are primarily and perhaps weakly based on talking to people in Mexico, although some research has seeped into my thoughts, I am sure:

1) Of those who answer yes to question 10.7 (do they borrow for business growth), I would expect an increase in 10.9 (how much they sell), and I expect no increase in 10.8 (expenses) unless these include interest expenses.

2) I would expect a reduction in responses 1 and 2 (spouse or family) for question 10.11 (who do you ask when you take out a loan).  I am not sure how I would interpret these declines though. Is it empowerment or just experience reducing the risk of the decision to borrow.

3) I would expect an increase in satisfaction with access to financial services (question 12.1 answer 3.

4) I would expect more stress and possibly depression, perhaps best captured by increases in 12.3 - 1 and 22.6

5) i would expect no change in schooling given the age of the women and the small number of children at home. 

6) I would expect no change in decision making (outside the specific issue of borrowing) given the short time frame of the study and also because these social structures require more than financial access to change. However, in the Locust of Control section, I might expect to see an uptick in 17.2 When I make plans, I am almost certain that I can make them work. because the experience of borrowing and paying a loan successfully may encourage women to have more self confidence.

7) i would expect the sample size to be too small to notice improvements in house hold characteristics such as roof, painting, floors. maybe a small increase in 4.1 but no change in any of the 4.2 more detailed answers. This is because women make so many decisions about how to use their very small marginal increases in income, I wont expect this one to show up statistically as important, even though for some people it is very important. 

 

By including 7 hypotheses, I may be increasing my chances of getting one right (exactly what the excercise tries to avoid), but I am truly interested in all these results!

Thanks for opening it up!

 

 

David, an important

David, an important difference between the two cases is that in Hyderabad virtually no formal microcredit existed before the intervention (if I have read the matter correctly) whereas in Sonora 44% of the population are already using formal microcredit services. Unless there is something compellingly different about the Compartamos product, or unless the informal financial sector is much weaker in Sonora than in Hyderabad, I would expect impact on household spending in Mexico to be less for this reason alone.

Metatesting

1. The critical outcomes to look for are 1) asset accumulation (in the form of durable goods, business equipment, savings, or years of schooling) and 2) reduced vulnerability (perhaps in the form of less severe hunger events).

2. The higher the ratio of respondents who are aware that IPA is specifically evaluating the impact of microcredit, or even more specifically, the impact of Compartamos (or Spandana, etc.) the more that respondents will overstate outcomes on any variable that these institutions represent publicly as central to their purpose (e.g. entrepreneurship, number of loans, size of loans, frequency of borrowing, assets accumulated, etc.). I hypothesize that even if IPA is attempting to keep these goals secret from respondents, it may not always be possible; and awareness may have a significant impact on testimonies even under a policy of non-disclosure.

Recovery of loans and sustainavility

The impact on mico credit pograms can be show in the recovery on loans given, it means capital and interest payed by the person who obtained the loan.  In micro credit programs destined to woman in recovery rates are high, 90% or more.

Other indicator can be the susteneability of the productive activities implemented with the money of the loan.  If the person is getting good net income (revenue less cost), it can be show that the financial resourser were adecuated invested.

No impact on household spending

At first I was going to hypothesize that the impact on household consumption would be the same as for group credit in Hyderabad (indistinguishable from 0) minus a factor that reflects the much higher cost of Compartamos credit.

 

However, I realized I was failing to consider that Compartamos credit could be substituting for other sources of credit that are even more expensive, thereby saving clients money.

 

So I hypothesize that the impact on household spending will be the same as in the other randomized studies, basically indistinguishable from zero.

Hypothesis: Microfinance offers the poor a better financial deal

In my new role as Senior Research Fellow at Freedom from Hunger, I am about to launch a blog-based “Evidence Project” to explore the evidence regarding the many hypotheses embedded in our “benefits diagram” (aka theory of change or logic model).  In effect, this is a crowd-sourcing approach to finding and interpreting evidence of many types from various sources.  The blog will launch on March 27 at www.microfinanceandworldhunger.org.  I invite IPA staff to participate a lot.  And I look forward to the evidence generated by the Compartamos impact research becoming important input to our Evidence Project.

This is not just a shameless promotion of the Evidence Project.  I have a complicated hypothesis to put forward for testing in your analysis of the Compartamos research database.  I’m starting the Evidence Project with the most basic question regarding the value of microfinance for the poor.  I’ll pose the hypothesis “Microfinance offers the poor a better deal” as a research question:  What financial costs does microfinance reduce for the poor?

This turns out to be a rather complicated question.  Where MFIs enter, we might assume they would displace the moneylenders as the preferred alternative to family and friends as lenders.   Even if a moneylender charges “only” 20% per month and the MFI charges as much as 5% (including fees and such) per month, the poor should be getting a much better deal from microfinance providers.  For a $100 loan held for one year, the borrower saves at least $180.  How can you beat that deal?

Well, the poor often prefer to take a loan for only one month, but the saving is still at least $15—a lot of money to save in one month.  However, MFIs usually do not offer one month loans, more often four months minimum, sometimes nothing less than one year.  The interest differential doesn’t look so persuasive, given that moneylenders offer one-month and even shorter loans.

But why would the poor go to moneylenders or microfinance institutions when they can get interest-free loans from friends and family?

From Portfolios of the Poor (p. 55), “The time, energy and emotional toll of borrowing informally appear to be global phenomena.”  Tim Ogden confirms and extends the conclusions of the Portfolios authors: “Looking at the financial services that the poor actually use the world over, from savings clubs to prepaid cards to microcredit to payday loans, there are a few common themes. Perhaps most important is that the poor are willing to pay relatively high explicit costs to offset implicit psychological and behavioral costs: shame, convenience, and temptation.” Imagine the stress of having to piece together the sum of money you need by going to people who know you socially!  The amounts and timing of these loans are unreliable, certainly not private, and subject to arbitrary and often unforeseen terms.

In contrast, MFIs are reputed to offer reliable, rule-bound and relatively impersonal, private and transparent access to credit (and increasingly saving services).  But they offer a truly superior deal to the poor only if they also offer the flexibility of terms and close-to-hand, any-time-of-day convenience of borrowing from friends, family, neighbors, shopkeepers and employers.  Moneylenders are reputed to offer more privacy and reliability than friends and family, along with the requisite flexibility and convenience—for a phenomenal fee which is bearable only for short periods.

Now we see that a “better deal” is not simply a matter of relative price.  The other dimensions we have to look at are amounts that can be borrowed or deposited, flexibility and reliability and transparency of terms, convenience and reliability of access, and privacy of transactions.  We have to compare microfinance (and similar) institutions, moneylenders and family/friends/shopkeepers/employers on these dimensions.

I’m not interested in saying “I told you so” or guessing the magnitude of the benefit or penalty of using MFI services.  I just want to know what the Compartamos study can tell us, if anything, about this and other fundamental hypotheses about the benefits and costs of microfinance services. 

Hypotheses for an Impact Study of Compartamos

1. Outcome measure: Is there substitution between use of Compartamos (Credito Mujer) and informal lending/savings mechanisms?

Comparing results to prior opinions

Re Comparing results to prior opinions, there are more interesting and important reasons for making these comparisons than simply enabling people to say "I told you so". Asking for public predictions before disclosing results is a way of assessing the immediate impact of research findings (so long as one has a profile of the public concerned, and one can ask some follow-up questions). For more on this, and related experiments by others, see Assessing the immediate impact of impact studies – using an online survey

Post new comment

The content of this field is kept private and will not be shown publicly.
Type the characters you see in this picture. (verify using audio)
Type the characters you see in the picture above; if you can't read them, submit the form and a new image will be generated. Not case sensitive.
*Required Field
Copyright 2014 Innovations for Poverty Action. All rights reserved.