Digital Finance Impact Evidence Summary

Between November 2016 and May 2017, we screened Digital Finance studies for contributions to the evidence on the effect of Digital Finance on individuals, households, and communities. This cumulated into the development of an Evidence Gap Map (EGM) for Digital Finance. At the time of writing, we have identified 40 Digital Finance impact studies from 21 countries that looked at various client level outcomes. We highlight that as an industry, we have made significant progress in establishing that digital payments and transfers are benefitting low income households, especially in Kenya. There is, however, a distinct lack of investment in evaluation and thus replication in different markets. Looking beyond digital payments and transfers, the digital finance community is not comprehensively measuring the effect of more sophisticated financial services (saving, credit, and insurance) on clients. Despite a growing number of Digital Finance products offered and the depth of the client base, evidence of client impact is lacking.

Introduction

An interactive Digital Finance Evidence Gap Map (EGM) was developed to provide a visual overview of the Digital Finance evidence landscape on client impact. The EGM summarizes the evidence on the impact of various Digital Finance products on several outcomes for low income clients, whilst also highlighting the direction of the effect, i.e., positive, negative, and no observed effect.

The input variables (rows) are organized into ‘Digital Finance and ‘Design & Delivery’. The Digital Finance element highlights the impact evidence of various Digital Finance products (e.g., a digital savings product).  The ‘Design and Delivery’ element enables us to highlight the impact evidence of interventions to drive further usage of these services (e.g., two-way SMS to improve saving behavior).  See screenshots below.

These two elements are layered to see their interplay (e.g., changes in savings behavior attributable to interactive SMS). The objective is to show, which Digital Finance product was studied, how it was designed and delivered, and what was the direction of effect.

Based on the literature that informed the development of the EGM, this snapshot serves as a summary analysis of the broader evidence-based mapping activity described in extensive detail on the website.¹

Summary of the effects of Digital Finance on low income clients

Between November 2016 and May 2017, through snowball sampling and mining bibliographies, we screened hundreds of studies. At the time of writing, we have identified 40 Digital Finance studies that looked at various client-level outcomes, of which 26 (65%) had a counterfactual. The remaining 14 (35%) are ethnographies, social science, action research, or case studies. We are tracking several impact studies that are ‘in progress’ and impact studies that were released after the initial literature review. These will be folded into the next version of the Digital Finance studies EGM.

This section presents the high-level insights from the Digital Finance studies impact literature review.

Sub Saharan Africa (SSA) highly represented

Twenty-one countries are represented in the EGM. SSA dominates the literature with 65% (n=26) of the studies. Within SSA, Kenya accounts for 35% (n=9) of the literature on Digital Finance impact.

Positive trend on number of publications

While the total number of studies is not large, we see a positive trend in the number of studies published each year. Not surprisingly, about half of the studies (n=19) were published in 2016 and 2015.

Emphasis on Digital Finance adoption as an outcome

Across the 40 studies, 72 information points were linked to 10 client outcomes of interest. Over a quarter (26%; n=19) of the outcomes focused on the adoption of a Digital Finance product. Increasing savings (15%), income (13%), income generation and response to shocks (both 11%) were the next most frequently evaluated areas. This information shows the researchers outcomes of interest per a particular Digital Finance product. The distribution of outcomes evaluated under each Digital Finance product is of greater importance when determining resource allocation. Digital credit, savings, and insurance all have significant gaps in evidence beyond product adoption; thus, we lack an understanding of how financial inclusion will fundamentally influence the lives of the low-income populations.

Positive leaning results for the broad Digital Finance sector but mixed at a product level examination

While the number of studies currently available is small, considering the diversity of available Digital Finance products, we see both positive, negative and no observed effect of various Digital Finance products across client outcomes. Overall, 77.8% (n=56) of the outcomes have come in as positive, 9.7% (n=7) as negative, and 12.5% (n=9) did not have any effect. Still, when examining each Digital Finance product by outcome and impact level, we see that the larger volume of digital payment and mobile money studies drove the positive results. We have little to say about the effect of digital credit, saving, and insurance products.

Digital payments and transfers are the most fully formed, evidence-based pathway to client effect

Across the 40 studies, 41 Digital Finance products were evaluated for various client level outcomes. Digital payments and transfers accounted for 54% (n=22) of the Digital Finance impact evaluations. When broad mobile money studies (studies that evaluated mobile money in a broad sense and did not specify the product only ‘mobile money users’) are included in this category, this increased to 71% (n=29) of studies. In 86% (n=19) of the studies coded as payments and transfers, mobile money was the mechanism used to make payments and transfers.

Under payments and transfer products, all outcome areas, except ‘healthy borrowing,’ have been evaluated. We see limited evidence that digital payments and transfers improve savings behaviors², though there was a suggestion of improved savings in a G2P Cash Transfer program in Mexico through access to ATMs cards³. Theoretically, the association of digital payments and transfers with savings behaviors is not strong. If we viewed digital payments and transfers as a means to increase access to quick cash, individuals would use them for their immediate needs; however, other products may be preferred when saving for a longer-term purpose.

Research has shown that B2P and D2P are cost effective, but there is limited evidence to suggest a broader use of digital financial services beyond the receipt of digital salary or cash transfers.⁴ However, there is evidence to suggest that digital payments reduce time and cost of travel.

Digital payments and transfers are frequently cited as the methods used to receive remittances in times of shocks, and research supports that this is the case for the East African Market.⁵ However, this was not the case in the Pakistan example, where digital payments have not taken off to the same extent.⁶ This highlights market specific effects of digital payments and transfers.

Income investing and welfare and income gains are highly populated with evidence, which supports that digital payments and transfers can result in improvements either through less leakages, direct income, or informal loans remitted.⁷

Sophisticated financial services are under-evaluated, and they have focused on measuring product adoption

Nine studies on Sophisticated Financial Services were identified. Seven examined savings, with only one study investigating credit and one study investigating insurance products.

Under digital savings, we learn much about product adoption;

1. Two-way SMS has the potential to boost the saving behaviors of clients.⁸
2. Products designed with minimal frills that are simple to access have seen successful uptake.⁹
3. Integration with existing customers who already know how to use the available services has shown positive adoption effects.¹⁰
4. Additional steps outside the deposit and withdraw cycle of savings may deter users.¹¹
5. Default contribution with additional employer contributions can improve saving behaviors of employees.¹²
6. User fees affect client uptake.¹³
7. Product design needs to carefully consider the use case for women.¹⁴

However, the potential impact pathway of digital savings is under-evaluated, as only one study looked beyond adoption and saving behaviors to evaluate the ability to respond to shocks and found no effect.¹⁵

Only two studies examined the adoption of one digital insurance product and one digital credit product. Here, we learn about how an investment in intermediated support can facilitate new technology for loan repayments¹⁶ and the potential of a freemium model to increase uptake of micro insurance products.¹⁷

The Digital Finance industry is growing, with GSMA reporting yearly growth in the number of live mobile products.¹⁸ The deficiency in evidence is stark in comparison to the growth.

Lessons from mobile banking provide a cautionary account of unintended consequences when changing from analogue to digital

Since many financial services for low income and rural populations are delivered in a group setting, digitization may then disrupt the existing social architecture, leaving its overall effect uncertain. From one study, we see negative effects on deposits, savings balance, borrowing, welfare, and income after the digitization of a saving group product.¹⁹ Much of these effects may be driven by weakened group cohesion and sensitivity to transaction fees and concentrated among members who lived near banking locations at baseline and had stronger connections to their microfinance groups. While positive effects on lowering the costs of transactions are clear, the unintended consequences were eventually unearthed. Without an impact evaluation, these would not have been revealed.

Conceptual clarity of studies

Seven studies examined mobile money, and three examined mobile banking. These are broad terms that may encompass multiple products, but the studies did not or could not specify the singular or multiple products used. This effects the utility of these studies as we are unclear on what Digital Finance elements were tested and how we can use the results.

Four out of the five studies on the longer-term outcomes of mobile money used a diverse set of definitions of a mobile money user. 1) Agent proximity, 2) ‘has mobile money on their own or another phone,’ and 3) ‘ever used’ mobile money are examples of the variety of independent variable descriptions. While sophisticated statistical methods are used to control for confounding variables, it is difficult to discern the contribution and understand why and how these effects may be associated with mobile money alone. Baumüller²⁰ reached a similar conclusion in her literature review of mobile phone enabled services and stated that “most studies do not assess impact in relation to usage. Rather, research tends to distinguish between users (or those with access to the mobile service) and a control group, and then compares impacts for the entire groups irrespective of usage patterns”. Access or ‘ever used’ mobile money are significant leaps to an active user definition and the benefits users may gain. These studies nonetheless provide insights and suggestions to the effect of mobile money, however, more defined measurements are recommended.

Hesitancy to show negative effects

A recent study on the effects of digital banking in the Philippines reported a negative influence of the digitalization of informal financial services.²¹ A study on the use of ATM cards to improve bank account use in Kenya found a significantly positive effect on male accounts but a negative effect on women’s account use.²² Seven studies on various types of accounts, e.g., D2P, B2P, G2P, P2P, and mobile savings, yielded null results on some of the studied outcomes.²³ While it is difficult to quantify, it appears that there is a hesitancy to report negative and unintended consequences, likely due to publication bias.

Implications for future research

Based on these high-level findings, we have indicated several potential directions for the future use of research resources that will address the gaps in evidence and contribute to a more robust understanding of client impact. We have classified three types of evidence gaps:

1. Foundational evidence

   : Evidence that addresses the foundational questions of the effect of various Digital Finance products, i.e., Does X Digital Finance product contribute to improved outcomes for X clients?

2. Evidence on optimal Digital Finance design and delivery

   : Nuanced evidence on the design and delivery mechanisms of various Digital Finance products. Ideally, this is incorporated with foundational evidence, i.e., Does X Digital Finance product with X, Y design and delivery mechanism contribute to higher adoption rates, greater outcomes for clients, and improved business models?
3. Evidence on Digital Finance service bundling: Evidence that enhances our understanding of the effect of product combinations or a suite of Digital Finance products, i.e., What combination of Digital Finance (and non-Digital Finance) products and services create greater outcomes for clients?

Recommendations for further research for each of these categories are provided below.

Foundational evidence

Focus on Sophisticated Financial services

Digital savings, credit, and insurance products are under-evaluated for client impact. Digital products and services for savings, credit, and insurance are growing, yet there is almost no evidence on their effects on income and welfare of the client. Any impact study that examines the effect of these services delivered digitally would add to industry knowledge. Advanced insights on the effect of sophisticated financial services can be obtained through understanding their effects on different populations or user groups, for example, women, rural and urban populations, older users, or vulnerable populations, such as refugees. Or, under economic activities, such as farmers or small business owners.

Example question: What is the effect of X digital savings or credit or insurance product²⁴ on X population? How did this effect occur?

Evidence on optimal Digital Finance design and delivery

Digital traits

What sets digital apart from analogue is the more nuanced ways in which digital insurance, credit, and savings can be designed and delivered. Currently, the focus on interesting design enhancements or delivery mechanisms that make Digital Finance unique is limited. For example, experimentation in gamification, chatbots, two-way SMS, smart interfaces, biometrics, pricing models, and others. Investing in studies that examine the vanilla Digital Finance with various toppings in design and delivery would fill in the gaps, bringing greater benefits to both clients and business models. CGAPs framework of digital attributes²⁵ is a useful structure to examine the effect of design and delivery mechanisms (as shown below). This type of research often focuses on improving product adoption, activity over time, and user satisfaction; thus, rapid A/B testing rather than RCTs would often be a more appropriate methodology when testing only the design and delivery mechanisms. Preferably, the effect on client level welfare outcomes would also consider both the Digital Finance product and the design and delivery mechanism.

Example question: What is the effect of:

1. Using advanced data analytics on client transactional patterns to tailor service offerings or segment clients?
2. Leveraging social networks within product design?
3. Developing smart and rich user interfaces appropriate for the target clients?
4. Integrating customized two-way real-time communication via SMS or chatbots?
5. Allowing for instant verification using GPS, cameras, or biometric data to verify identity or location?
6. Using embedded GSM technology to track movement and usage of movable assets?

Example question: Does A, B, C, D, E, F design and delivery approach increase product adoption and activity over time? That is, do they improve savings behaviors (saving products), risks taken (insurance) and repayment rates (credit) and among what populations?

Example question: Is the effect of Digital Finance products with enhanced design and delivery mechanisms on client outcomes greater compared to the effects those without enhanced design and delivery mechanisms? For what populations?

Evidence on Digital Finance product bundling

Digital Finance bundles

What are the effects of using a combination of Digital Finance (savings, insurance, credit, and ePayments), as opposed to using only one, some, or no Digital Finance? Additionally, we are seeing that more providers offer Digital Finance products bundled with Digital Information Services (DIS), which contain relevant content for the target clients, for example, financial information, agricultural information, or business management. Bundling and unbundling requires significantly more complex studies and resources, but it could have far reaching implications on how we approach financial inclusion developmentally.

Example question: What is the combined effect of using a wide array of digital savings, credit, and insurance products on client outcomes? Which combination of products is most influential? For which populations?

Example question: Can combing Digital Finance with DIS enhance the effect of Digital Finance products? For which populations?

Other considerations in future impact research

Consider a focus on impact pathways, questioning not just what changed but how it changed

Digital Finance is not a homogeneous category; there is value in untangling the effect of each Digital Finance product and deepening our understanding of not only the changes in the lives of low income users that each of these products can catalyze, but also how the users experienced that change. This can result in a more nuanced understanding of how various products may contribute to various outcomes, allowing us to begin to form the impact pathways for each Digital Finance product based on emerging evidence. This will provide the sector with the capacity to delve into the details of the market and social conditions in which the Digital Finance product was or was not successful in catalyzing change.

Awareness of potential negative effects in research design

Consider research design that also actively explores the unintended or negative consequences of digital saving, insurance, and especially credit products. For example, concerns about the design of digital credit solutions are growing not only in terms of high interest rates,²⁶ but also in terms of the growing number of clients who are being blacklisted by credit bureaus for outstanding loans.²⁷ This information is crucial to inform better product design.