2013 in review

The WordPress.com stats helper monkeys prepared a 2013 annual report for this blog.

Here’s an excerpt:

A San Francisco cable car holds 60 people. This blog was viewed about 810 times in 2013. If it were a cable car, it would take about 14 trips to carry that many people.

Click here to see the complete report.

What Big Data really brings to the table

What Big Data really brings to the table

 

The term ‘Big Data’ has become incredibly miss-used over a very short life. There are many ways in which it can be defined but, forget 3, 4 or 5 V’s for the moment and any other ways to categorise ‘Big Data’ and figure that instead of a type of usage of certain types of data, it’s more of a different way of thinking, it’s a new era in data processing if you like. Quite frankly the old tools based BI accompanied by the endless and often moronic discussions of Data Warehouses vs Data Marts etc was getting incredibly boring after 20 or so years. Lots of money was being spent, some wisely, some not and just a handful of vendors were making the whole market. Often as not, companies spent loads of money on Data Warehouse platforms because they were scared not to!

Now something has happened that is completely different. Do we have vast new sources of data all of a sudden?  I doubt it. Are statisticians really the saviour of our planet? I doubt it. Do we need 900000 extra Data Scientists? I doubt it. Is statistics suddenly sexy? I think not. What has happened is that at last ther is a deep ‘feeling’ that analysing the data spawned from all sorts of sources is a valid and useful thing to do. At last, business people seem to have woken up to the facts that there is help out there to support or otherwise, there decisions and this is at the heart of Big Data. It’s a movement, a realisation that actually knowing something (OK, with a degree of certainty only) is far better that simple gut feeling. And off we go……..

Me? I’m excited. I wrote a book some time ago which pretty much outlined my disillusionment with BI over the last 20 years and now I’m excited again. Can we use data to impact and improve our daily lives as people (consumers) on this planet. I think we can but it won’t happen quickly and it won’t happen without a lot of discussions on privacy etc, etc.

So to be clear. For me Big Data represents a time in which we have suddenly (and it is sudden) become aware of the value in data. It’s not about a highly over-hyped Hadoop or any such technology, it’s about people, new opportunities and new realisations

The Role of Business Intelligence and Data Mining in Credit Management

So, what is the role of a Business Intelligence in this world of credit assessment?

Let’s imagine that we had the following items of data about our customers over a history of thirteen months:

• Ø CDRs for incoming and outgoing calls
• Ø Invoice details
• Ø Payment details
• Ø Personal attributes – gender, date of birth, address etc.
• Ø External demographics,  credit scores etc.

By running complex queries (neural networks and various statistical operations) over this history we can find answers to the following questions based on people that we know have already used, or are using, our own services:

• Ø What behaviour consistently leads up to someone defaulting on payment?
• Ø What types (segments) of customer tend to default on payments?
• Ø Are there significant patterns in defaulting for individual customers?
• Ø Is defaulting in payment correlated to:
  • unusual increase in line usage?
  • an unusual and sudden amount of international calls to a specific location?
  • time of the year?
  • other economic happenings?

Based on this type of analysis and some simple profitability modelling we can assess the money actually lost through non-payment and compare it with the costs of various treatments. We might also start to allow different segments of our customer base to default under certain circumstances, safe in the knowledge that they will likely return to profitability in the near future. Similarly we can start to predict the likely credit worthiness of specific potential customers and modify our marketing strategy by the perceived risk, setting up more sophisticated contractual agreements where risk is counterbalanced with assets.

 

Exploding Some Uninformed Assumptions

Let’s look at a point made earlier a little more carefully – A bad customer to one company can be a good one for another.

Take the real case where two mobile telephone provider companies in a single advanced country compete with each other head-on. One company has built a reputation as the high-quality (and expensive) provider whilst the other provides cheaper services to a different mass market. What if they both used the same credit model to filter new customers? How long would this carefully engineered two-tier strategy last, and what would be the consequences as both companies try to attract the exact same people?

The answers should be obvious but the fact is, of course, that few telecommunications companies do any rigorous credit assessment for new customers or those existing customers buying new products, but when they do, we can be assured that one model for all operators will bring nothing but grief for operator and consumer alike. We quickly learn that credit models must be in tune with the acquisition and retention strategies of each specific provider.

Let’s examine another phenomenon – A person who defaults on a few payments may be a very good long-term customer.

In a mobile world, as we have already mentioned, it is very difficult (and dangerous) to screen customers out completely based on credit worthiness prior to providing them with any services at all. This is why traditional credit scoring is not the answer to most such problems. What we want is to provide service and then monitor usage and payment history over a time period (perhaps three to four months) to determine behaviour and then treat any subscriber showing ‘abnormal’ behaviour. Over this ‘bedding-in’ period we might:

• Ø Discover those people who have received mobile phones as presents and who really have no intention of payment because they don’t value the service (or maybe even suddenly realise they have to pay).
• Ø Discover those people who obviously didn’t understand that mobile telephony can be expensive and are horrified at the bills, finding suddenly that they can’t or won’t pay.
• Ø Discover those people that didn’t read the contract note carefully enough to understand that they still need to pay line rental even when they have given up using the phone.
• Ø Discover those people who have a phone only for incoming calls.
• Ø Discover those people who only use the phone for emergency use.
• Ø Discover those people who habitually pay bills late (or quickly).
• Ø Discover those incidents of non-payment that were caused by some malfunction in the service provided (failures of the billing system for example).
• Ø etc., etc.

Getting back to the general threat, last post but one

Why Default?

It is prudent, therefore, to understand why people (individuals) default on payments before making rash judgements regarding how they should be treated. There are many, many reasons and all are understandable given the facts.

For example, the simple task of paying bills turns out for some people to be a complex issue, often subject to arbitrary and elusive rules. We should consider the following:

Only a small percentage of credit risk can be predicted using current static systems.

Today, credit risk assessment is largely the prerogative of specialised credit agencies that provide companies with either fixed algorithms for filtering customers requiring credit at the point of contact, or provide fixed scores computed over minimal attributes for the same purpose.

Thus, if I elect to purchase a product without using cash, the agent I deal with might already have a credit score of mine supplied by an outside agency (either on-line or dial-up) or may put me through some mini interrogation previously defined to gain answers to key credit worthiness questions – the result is the same, I either pass or fail and the goods are mine or otherwise.

Such systems cannot be considered ‘state of the art’ for many simple reasons:

• Ø In reality, credit authorisation is a time-consuming, expensive and irritating feature of modern life.
• Ø Web shopping will make traditional credit authorisation prohibitively cumbersome.
• Ø Credit agencies do not have much of the data pertinent to credit authorisation, much of what they do have is out of date
• Ø Critical data, although often available, is not used by third-party credit companies.
• Ø All credit models are fundamentally the same – very little differentiation is possible even though many competing companies have their own unique customer profiles.

Subject: Are you Member of The Big Data Institute (TBDI)? Join now whilst Free Basic Membership lasts!

Subject: Are you Member of The Big Data Institute (TBDI)? Join now whilst Free Basic Membership lasts!

 

Did you know that there is now an official body for Big Data, Data Science and Advanced Analytics?

 

Just a quick heads up to tell you of the newly created Big Data Institute. I think it’s a valuable concept and intend to take an executive position in the group to ensure that it remains relevant to EMEA.

The Big Data Institute (TBDI) is a non-profit voluntary organization for Big Data Analytics, Data Science, Advanced Data Visualization for executives and professionals worldwide. It will provide Free and Premium membership to give access to sponsored content such as research, Webinars, and white papers.

This is a voluntary organization and represents a unique opportunity to make an impact in this leading technology by sharing ideas across the globe

TBDI is now having a promotion for FREE Basic Membership ($100 value) till 4/15/2013.
Membership information: http://www.the-bigdatainstitute.com/Membership.html

Please visit the website and join up ASAP. If you have problems, try again soon

You should also join TBDI – The Big Data Institute LinkedIn group to know more!
https://www.linkedin.com/groups?gid=4900898&trk=hb_side_g

Obviously the group is new and so activities and web content is sparse at present but I ask you to join up and contribute so we can get going. We have big ideas and have already signed a Media Agreement with IQPC for Big Data Conferences and Summits.
a. http://www.bigdatahealthcaresummit.com/MediaPartner.aspx
b. http://www.bigdatamarketingforum.com/MediaPartner.aspx

Soon, TBDI will also release a certification program.
CBDP – Certified Big Data Professional
CDSP – Certified Data Science Professional

Our full website is coming soon with a Data Science Book of Knowledge and Big Data Methodology. Stay tuned!

Thanks!
TBDI Team

(Jon Page)

Follow us on Twitter @BigDataBody
Follow TBDI company on LinkedIn @ https://www.linkedin.com/company/the-big-data-institute?trk=hb_tab_compy_id_3009704

BI in Credit Management

There are many, many applications for BI in the world today. They are very varied, and to show another I’m going to pick on the sensitive subject of credit worthiness.

 

Credit Worthiness Myths

Three of the most talked-about phenomena in the telecommunications industry today are fraud, churn and credit worthiness. Whilst Data Warehousing is often seen as a potential solution to the first two, it is rarely seen as a vehicle for tackling credit worthiness. This post seeks to address this and to show that a Business Intelligence environment holding historical and detailed data regarding usage, billing and payment information is the ideal place to detect and predict credit exposure. In fact, we can see that the three phenomena mentioned are all part of a bigger picture. Ability and willingness to pay bills is a key factor in determining churn – people committing fraud usually churn very quickly, and of course, anyone deliberately using services with no intention of payment is in fact committing fraud. It would seem that a solution to all three might lie in the same area.

Let’s start by examining some misconceptions about credit such as:

• Ø All ‘poor’ people have a propensity to default on payments.
• Ø Certain types of people are born with the inherent propensity to default on payment.
• Ø Once a credit risk, always a credit risk.
• Ø All people who default on payments are ‘bad’ customers.

The perceptions above are very common in a world that doesn’t understand credit worthiness except in the most rudimentary terms. From experience we find that the truth is a little different. In fact:

• Ø A great deal of the credit risk a company must contend with is a calculated result of sophisticated fraud – nothing to do with people’s individual propensity to pay bills at all.
• Ø People go through phases in their lives when they move toward and away from the propensity for payment default. Whilst it may be true that a hard core of people will never default because of a strict moral code, one simply cannot pay if funds are not available, no matter what ethics apply.
• Ø Similar to the above, we often find that as students and young adults, defaulting on payments doesn’t seem to have any real consequences (moral or otherwise) and so such people are often very lax when making payments, whether funds are available or not. As age brings experience, such people usually become model and profitable customers.
• Ø Traditional credit risk systems provide processes to deal with people after the risk is realised. However, the real state of the art is to predict who will default and take action before it happens.

Illustrating BI Complexity

Let’s illustrate the complexity with another example similar to one I’ve used before, in which I will once again imagine myself in the marketing department of a mobile phone provider, where I am surrounded by a pretty spectacular BI environment.

This is how my thought process goes:

Already I have a report on the people who ‘churned’ last month (swapped to a different operator), I know who they are, what products they use and when they use them, but I want to know why they churned.

My suspicion is that a new competitor has just started offering a heavily discounted set of tariffs and so I analyse just how much my churned customers could have saved in the last month using this new tariff, and I discover it’s significant but not threatening.

My next task after completing my ‘What’ and ‘Why’ tasks is to try to predict the future. Using advanced analytics and the known profiles of the people that have already churned, I can make a go at predicting who will churn next month, and so out pops my list of likely churners. Now this is a complex task and is never wholly correct, but it’s better than nothing by a long way.

Now let’s make this individually focused so you can see the difficulty and the opportunity. Let’s say that I’ve identified a likelihood that one of my customers, a Jon Page, is likely to churn next month. Now in order to keep Jon it’s very probable that all I have to do is offer him a cheaper tariff, but if I do this, and Jon accepts, what will I have done to Jon’s profitability? The problem here becomes even more complex because I have to do some real thinking. Firstly, do I want to stop Jon churning? If he is in the lowest or non-profitable segment then I should be happy to lose him and so I need take no action. However, if Jon is in the ‘growable’ segment then I must find a way to keep him whilst not reducing profitability over his lifetime. This remains a difficult task involving many cross-relational predictive models and lots of data.

So, let’s imagine that I want to keep Jon without reducing the money he spends on his calls. What can I do? Well, by researching his call behaviour I can identify that he makes many calls on Sunday afternoons at a cheaper rate, so maybe I could offer him these specific calls for free. It won’t make much difference to overall revenue but it might really impress Jon and he might even make up the shortfall by making more or longer calls at another time.

Maybe there’s another way. From the detailed call data I have, I can figure out that in the last three years Jon has holidayed in Malta, so maybe I could offer him a one-off discount voucher for a week’s holiday in Malta for two. At a cost of maybe £200, once and only once, maybe Jon would be so impressed he’d stay a customer for life and he’d certainly tell all his mates how good his provider is!

So, a lot of analysis has to be done, indeed a great deal of thinking and some passion are needed to win.

2012 in review

The WordPress.com stats helper monkeys prepared a 2012 annual report for this blog.

Here’s an excerpt:

600 people reached the top of Mt. Everest in 2012. This blog got about 2,100 views in 2012. If every person who reached the top of Mt. Everest viewed this blog, it would have taken 4 years to get that many views.

Click here to see the complete report.

some interesting observations on the BI market

I thought you’d be interested in an article on the market opportunity for business intelligence from Software Advice.. Check it out: http://blog.softwareadvice.com/articles/bi/is-the-market-opportunity-understated-1083112/.

The article describes key trends in business intelligence software and the understated opportunity for BI

 

I like a couple of quotes in particular:

‘In the future, analytics and caring about data will become a part of everyone’s job,” says Caleb Poterbin, head of marketing at analytics software provider Chartio

Marketing is heading for a revolution me thinks….

‘It’s not necessarily always about deep-level analysis,” says Jeffrey Vocell, co-founder of Trendslide. “It’s about that snapshot.”

Well this comes from a snapshot type of company but in truth, the more we get fed this ‘big data’ non-sense the more we are likely to forget that most BI is visualized through extreme summarization and not always with any link to the underlying detail.