I was invited to present at the Federal Reserve Bank of Philadelphia’s Payment Card Center conference, Consumer Financial Protection Regulations: How Do They Measure Up?. Held last week in Philadelphia, this excellent two-day event was tailored around different aspects consumer financial protection to help advance the discussion of targeted design and outcome measurement as central features of public policy in the area of consumer financial protection. I was part of a panel of professionals who discussed the challenges of analyzing consumer complaints from the Consumer Financial Protection Bureau (CFPB). I focused specifically on the complaints that the CFPB are gathering from consumers regarding credit cards to shed light on the use of the CFPB credit card complaint data and the quality of these complaints.
Consumer Financial Protection Bureau Complaint Database
The central mission of the Consumer Financial Protection Bureau (CFPB) is to “make markets for consumer financial products and services work for Americans — whether they are applying for a mortgage, choosing among credit cards, or using any number of other consumer financial products.”
The CFPB credit card complaint database reflects complaints received by credit card consumers. While the CFPB does not verify the accuracy of all facts alleged in these complaints, they do take steps to confirm a commercial relationship between the consumer and the identified company/issuer of the card. While the overall credit card complaint database includes data from June 1, 2011, the only data available to the public includes complaints starting on June 1, 2012 (it is expected that prior complaint data will be made public later this year). In the public database, each submitted credit card complaint is described by nine (9) variables:
- Method of complaint submission (Email, Fax, Phone, Postal mail, Referral, Web)
- Date complaint received (day/month/year)
- ZIP code of consumer
- Issue (type of credit card complaint – there are 33 issues from which to select)
- Date the complaint was sent to the company/issuer of card (day/month/year)
- Company/Issuer
- Company response (7 types including such as closed, closed with explanation, closed with monetary relief)
- Whether the company response was timely (Yes or No)
- Whether the consumer disputed the company response (Yes or No)
Table 1. Frequency of specific complaints for the first three months of CFPB database (July 21, 2011 through October 21, 2011) and complaints from June 1, 2012 through August 29, 2012.
For the present analysis, I downloaded the credit card complaint data from June 1 through August 29, 2012. The total sample size for this data set was 3978 (June = 1671; July = 1494; August = 813).
Criteria for a Good Customer Metric
Consumer complaints, a specific kind of customer metric, need to be evaluated along four criteria. A customer metric is good when:
- The metric is supported with a clear definition of what it measures and what is does not measure.
- There is a clear method of how the metric is calculated, including all items and how they are combined.
- The metric has good measurement properties; that is, there is good reliability and validity evidence regarding how well the customer metric measures what it is supposed to measure.
- The metrics are useful in helping drive real internal changes (e.g., improved marketing, sales, service) that lead to measurable business growth (e.g., increased revenue, decreased churn).
For a more, in-depth discussion about these four criteria, please see Four Things you Need to Know about Your Customer Metrics.
Consumer Financial Protection Bureau Credit Card Complaints
Let’s evaluate the CFPB complaint metric using the above four criteria:
Table 2. Data for Credit Card Issuers – Complaints from CFPB complaint database; Number of credit cards; Normalized complaints; and JD Power and Associates Ratings from their 2012 US Credit Card Satisfaction Study.
1. Definition
The definition of a CFPB complaint is clear and straightforward. A CFPB credit card complaint is an expression of dissatisfaction or discontent with the credit card experience, a negative customer experience. While a complaint likely impacts customer loyalty (e.g., likely to recommend, buy again), a complaint is distinct from customer loyalty. When submitting complaints, consumers can categorize their complaints into one of 33 different categories. As such, each credit card issuer can have 33 different complaint metrics, one for each type of issue/complaint (e.g., Billing disputes, APR or interest rate, Closing / Cancelling accounts, Credit reporting).
Figure 1. Relationship between total complaints (normative) and satisfaction ratings across 10 US banks (r of rank order = .66)
2. Calculation of Complaint Scores
The calculation of complaint scores involves summing the number of complaints associated with each variable. Summed scores can range from a low of 0 (no complaints) to a high of N (the total number of complaints).
Comparing Banks
Because total number of complaints is impacted by the sheer number of cards in circulation, normative complaint scores were calculated for each bank by dividing the complaint scores by the total number of card holders for that specific bank / card issuer (based on the Nilson Report, a card-focused trade publication). These data were from February 2012 and include numbers for only large card issuers.
Figure 2. Relationship between billing dispute complaints (normative) and satisfaction ratings across 10 US banks / card issuers (r of rank order = .72)
3. Measurement Properties
Reliability
I examined the relationship of the frequency of complaints for specific issues across time 1 (first three months of CFPB complaint data – see Table 4) and time 2 (June 1 through August 29, 2012). The data are in located in Table 1. The correlations between the frequency of different complaints across time was .89, suggesting that the relative frequency of complaints remains stable over time. Top complaints in time 1 (e.g., Billing disputes, APR or interest rate, Identify theft / Fraud / Embezzlement) are top complaints in time 2 and less frequent complaints in time 1 (e.g., Bankruptcy, Balance transfer fee, Overdraft fee) are less frequent in time 2.
Figure 3. Relationship between APR or interest rate complaints (normative) and satisfaction ratings across 10 US banks / card issuers (r of rank order = .34)
Validity
There are three different lines of validity evidence that help show what the customer metric is measuring, I will present criterion-related validity evidence regarding the complaint data. For top US banks / card issuers, I examined the correlation of their normative scores with an independent ranking of banks / card issuers. JD Power and Associates conducted a credit card customer satisfaction survey (data collected in June 2012) on large card issuers. I correlated the JD Power and Associates satisfaction ratings with the CFPB normative complaint metrics across 10 US banks / card issuers (for all complaints, Billing disputes and APR or interest rate). Table 2 contains these data. These 10 banks were selected because they had ample data from both the CFPB and the JD Power and Associates credit card customer satisfaction survey.
The correlations between the satisfaction ratings and the three normative complaint scores are -.44 (all complaints), -.57 (Billing disputes) and -.35 (APR or interest rate). That is, banks with higher complaint scores also received lower satisfaction ratings. To illustrate these relationships, I plotted the banks on their ranking of frequency of complaints (1 – low complaints; 10 – high complaints) against the ranking of banks based on their satisfaction ratings (1 – high satisfaction; 10 – low satisfaction). These figures appear in Figures 1 through 3. Based on these analyses, we see that billing dispute complaints, compared to APR interest rate complaints, appear to be more strongly associated with customer satisfaction.
4. Usefulness
Figure 4. Company Response to Specific Types of Issues
I can see some practical applications of the CFPB complaint data. Specifically, these data can help banks track how they respond to different types of complaints and understand how their handling of complaints impacts customer satisfaction (see Figure 4). The data show that banks respond to late fee complaints primarily with some form of monetary relief to the consumer while other types of complaints are dealt with only an explanation.
Summary
The banking industry is opposed to publishing CFPB data on credit card issues; the American Banker Associations’ spokesperson, Carol Kaplan stated, “Until somebody has had a chance to sort though it and figure out what’s valid and invalid, it’s a gossip column.”
The current analyses, however, suggest that the CFPB credit card complaint database provides clear, reliable and valid information for banks about their credit card practices. The frequency of different types of complaints are fairly stable over time. Additionally, the normative CFPB complaint scores are related to credit card customer satisfaction ratings (from an independent source); banks with better complaint scores (lower number of complaints) receive higher satisfaction ratings compared to banks with poor complaint scores (higher number of complaints).
Additional Research Needed
While the current findings provide some evidence regarding the reliability and validity of the CFPB credit card complaint database, more research is surely needed to validate the current findings. Additionally, more research is needed to understand the usefulness of these complaint data. Some potential areas of future research including the application of text analytics to CFPB verbatim comments. To be sure, while not provided in the public database, verbatim comments from customers are given to the specific bank who issued the credit card. Open-ended comments from customer surveys are commonly used to identify specific ways to improve the customer experience. Perhaps these CFPB verbatim comments could provide some additional insight about how banks can improve their practices around credit cards. Does customer sentiment behind the complaints provide useful information about the customer experience? Do improvements in the credit card experience correspond to improvement in sentiment scores?
Also, it would be interesting to study different types of complaints. The current analysis showed that billing disputes might be more closely linked to customer satisfaction compared to APR and interest rate complaints. As/If billing dispute complaints decrease over time, do other types of consumer complaints of credit cards become more important in predicting customer satisfaction?
Finally, based on a very informal polling of the audience, I found that a few bank representatives indicated that they incorporate their CFPB complaint data into their company’s formal customer feedback/experience management program. I would be interested in understanding how banks are using these CFPB complaint data as part of their larger program to improve the customer experience.
I would love to hear what other research ideas people have regarding CFPB complaint data.
