Four things to be aware of in the search for digital finance impact

In previous insight pieces on savings, credit, and, Person-to-Person (P2P) transfers, we synthesized what we learned from studies in the Digital Finance Evidence Gap Map (EGM) using a product lens. However, the product lens is just one perspective, and the digital finance impact landscape is more varied and layered than this. Here we share four factors that digital finance researchers should consider when testing the impact of a digital finance product.

Figure 1: Source of variance on impact

1. Digital finance is not one thing

Digital finance includes dozens of diverse products, and researchers need to disentangle each digital finance product from the broader category in order to deepen the digital finance community’s understanding of what changes each of these products catalyze in the lives of low-income users. When the 60 product studies within the Digital Finance Evidence Gap Map (EGM) were disaggregated, we saw vast differences among the products that have been tested to date. P2P payments and transfers account for 30% (18) of the impact studies, savings for 18% (12), and credit for 12% (7).

Figure 2: Number of digital finance products represented in the Digital Finance Evidence Gap Map

Results from studies on P2P transfers cannot contribute to our understanding of the impact of digital credit. Indeed, every product comes with its own theory of what changes as a result of interacting with it. To advance knowledge in the digital finance community, we need to identify the gaps in product impact knowledge and allocate resources to begin correcting it.

2. Product design and delivery varies

Digital finance products can be and are designed and delivered in many different ways. The difference with digital is the opportunity to enhance services, and hopefully outcomes, by integrating design and delivery elements. An analysis of the EGM confirms that product designers use a multitude of innovative digital, and non-digital, design and delivery mechanisms. The EGM shows that across the 60 products studied, 28 different design and delivery features were observed. On average there were two design and delivery mechanisms described per product in the EGM.

Figure 3: Number of design and delivery mechanisms used across 60 products in the Digital Finance Evidence Gap Map

This means we might expect different client outcomes from digital credit products with a mobile learning component versus digital credit products without. Even within the same category of digital finance products, how they are designed and delivered may differ, and these differences result in variance in user outcomes.

However, most studies are not engineered to show whether the driver of change was (1) the way they recruited clients, (2) the training content, (3) the price of the SMS, or (4) some combination of these elements. These design and delivery choices typically become a single data point in a study so that we learn about the sum of the whole, rather than the sum of the parts. This is fine for deployments but a challenge for meta-analysis and impact assessment unless these variations and distinctions are made explicit.

3. Markets vary

A product owner could test a similar product in two different countries and find that client outcomes diverge. Regulation, infrastructure, capital, and, of course, the client’s ability and need to engage with the product vary across markets. The EGM represents 24 countries, but Kenya accounts for over a quarter of these impact studies (27%) and East Africa as a whole for over half (52%).

Figure 4: Number of countries represented in the Digital Finance Evidence Gap Map

Unsurprisingly, Kenya, the site of much innovation in digital finance, is the most studied country. But we must acknowledge that we simply know less about other regions and countries and that much of our learning comes from a particular region. So, before we transfer learnings from Kenya or Tanzania to other contexts, we must carefully consider the various social, economic, and cultural differences involved.

4. Clients vary

We could also test the same product in the same country with different client segments and find different results. One question to ask is, with which population segments are we testing various digital finance products? From a review of the 60 products in the EGM, the user sample has been concentrated on a mix of urban and rural clients (30%; 18), followed by exclusively rural clients (22%; 13). Fewer studies, 12% (7), have focused on smallholder farmers and, 8% (5), on women.

Figure 5: Number of client categories tested

With whom we choose to test our products matters. Just as a specific market focus can limit our generalizability so can the client sample. This is not necessarily a bad thing: in the context of digital finance, a number of outcomes are unlikely to be experienced equally by all clients. Often, some people benefit more, others less. But we only know when we test for it. See FiDA’s Snapshot on How do advances in digital finance interact with dynamics of exclusion? for more details. A quarter of the studies in the EGM looked for, and highlighted, differential effects on some level within a market based on gender, location, income, or education as described below.

Cases in which the marginalized benefit more from a digital finance product

In a study of M-Pesa in Kenya, results showed that M-Pesa users did not resort to reducing consumption as a coping mechanism when they faced a negative economic shock. Further, the effects were shown to be more evident for the bottom three quintiles of income distribution than for the top.
A study on mobile money in Burkina Faso found that individuals living in rural areas were three times more likely to save with mobile money than those in urban areas. Additionally, women were six times more likely to save with mobile money than men, and less educated individuals were four times more likely to save with mobile money than higher educated individuals.
A Kenyan panel study on M-Pesa reported that mobile money access reduced both extreme and general poverty and that the effects were more pronounced in women—women being affected more than twice as much as the average.
A study in Tanzania examined the impact of P2P transfers in the event of a [weather related] shock. They found that the benefits of other people using mobile money in the community come largely from rural areas where, with 1/3 of the village using mobile money, household benefit by a 10% increase in consumption. In urban areas this effect is only 3% and not significant.

Cases in which the marginalized may benefit less from a digital finance product

There are also cases in which the benefits of digital finance accrue to high status, high skill individuals, rather than to marginalized populations.

CGAP partnered with the Busara Center for Behavioral Economics and Jumo KopaCash—a mobile money marketplace that offers digital credit—to measure client responses to various repayment reminders. The messages’ results varied by demographic with positive effects on repayment for male borrowers and negative effects for female borrowers.
In a panel survey looking at the savings behaviors of M-Pesa users and non-users in Kenya, 65% of M-Pesa users reported having some savings, compared to 31% of non-users. The results showed that those most likely to have savings are male, live in rural areas, and have higher levels of education and income. In terms of the value of savings, those who registered with M-Pesa save 12% more than those not registered. Furthermore, the survey showed that those who are male, urban, and have higher levels of education and income tend to have more savings.
A study on the effects of ATM cards on bank account use in Kenya found that while ATM cards increased account use among male-owned and joint accounts, a negative impact was found on female-owned accounts.The hypothesis was that household pressures to share savings drove women to stop using their accounts when an ATM card reduced the costs of accessing their money.
Testing the effects of a roving Point of Service (PoS) on savings account use in Malawi, researchers observed differential levels of impacts based on wealth levels. For households at the top of the wealth distribution, an increase in savings services was associated with less reliance on distressed asset depletion to cope with economic shocks. However, the effect was the opposite for those at the bottom of the wealth distribution.

These observations offer the digital finance community a refined understanding of impact by determining the conditions under which impact is present, or stronger or weaker. They also underscore the need to examine how a digital finance product interacts with and affects various excluded groups.

The takeaways

Digital credit, particularly insurance products, are under-evaluated for client impact. While a growing community of practitioners are working to address gaps in credit knowledge (for example the Digital Credit Observatory, CGAP, FSD Africa), this is not yet the case for insurance. Digital insurance, a nascent product, has primarily, and understandably focused on business model testing. However, as a starting point, qualitative work should begin examining the insurance impact theory.
Much research has looked at what does and does not work in terms of product configurations and campaigns, but few impact studies are engineered to pinpoint whether the recruitment strategy, pricing, messaging, or something else that contributed to a given change.While thousands of choices go into product design and delivery, they typically become a single data point. Without understanding this nuance, it is difficult to discern exactly what we are learning.
The learning on digital finance impact is concentrated in East Africa. While this knowledge may be transferable, there is a notable absence of studies that focus on West Africa and Southern Africa.
Digital finance studies that have examined impact on more excluded groups show that impact is not homogenous. While a few studies specifically define a group of people, we lack research that disaggregates client segments. Acquiring this knowledge would advance our learning on product design and user needs.

Although these nuances may seem obvious to some and overwhelming for others, it is important to recall that while the impact landscape is complex, the studies themselves do not have to be. There is more than one way to gather impact insights, and if the digital finance community is going to be inclusive, we must ensure that we are expanding the conversation on impact to include, in an appropriate way, voices and signals beyond those that can be gathered by experimental methods (See FiDA’s Approaches to determining the impact of digital finance programs for more details).

To continue building the evidence base, the broader community needs to be more involved in contributing insights so that we can have active conversations about impact. Every study that targets a gap— in a product, a market, a design and delivery mechanism, or a client segment—brings the digital finance community closer to understanding the various impacts of different digital finance products and services. Developing an evidence base is like building a puzzle, no individual piece can reveal the picture, but, bit by bit, the picture will emerge as new pieces are added.

FiDA is publishing a series on insights derived from an analysis of the latest Digital Finance Evidence Gap Map (EGM) update. This is the fifth blog.

Previous blogs:

The studies in the EGM represent our best knowledge of digital finance impact insights. New studies are ever emerging and thus the EGM will continue to evolve. If you have questions about the EGM, are interested in discussing research priorities, or know of relevant digital finance impact studies that meet the inclusion criteria, please contact ideas@financedigitalafrica.org.