Data Mining "Three Points" Handbook

As a senior consultant of IBM's global business intelligence solutions in China, Zheng Xiaojun was asked when he was in contact with target users: "I want to classify customers now, should I use data mining?" Zheng Xiaojun Ask the other party: "What kind of customer division do you want to do? If you only divide according to the traditional social background of customers, then there is no need to use data mining; if you want to divide according to customer behavior patterns, you can use data mining. "

This scene is currently reproduced in many Chinese companies. On the one hand, they have begun to recognize the significance of data mining and have a positive attitude towards it. On the other hand, they are not very clear about what kind of foundation the data mining should have, what problems should be paid attention to in the implementation process, and how to obtain the ideal return on investment.

Data mining is a high-level business intelligence application, not a one-shot project. Enterprises should at least figure out the following three points: the starting point, the difficulty, and the key point.

Starting point: building a data warehouse

What information technology foundation does an enterprise need to have in order to apply data mining technology? The ideal starting point is to build a data warehouse.

This data warehouse should store all customer data, and there should also be relevant data on market competitors. . The data for data mining should come from the data warehouse. The database used can be a database from various markets: IBM, Oracle, Sybase, NCR, etc.

Some enterprises have come into a misunderstanding: before the data warehouse is built, they directly apply data mining applications. After spending a lot of human, material and financial resources, the company found out why this project is so difficult to do. As a result, there is a high probability of abandonment, and the project is aborted.

This result is due to seeing only the attractive prospects of data mining and underestimating the difficulty of preparing data. The workload in the early stage of data mining is very large, and almost 80% of the work is preparing data. Data must be integrated, extracted, cleaned, transformed, loaded, and when all data is ready, data mining can be effective.

There is a well-known saying in data mining that "garbage in, garbage out", which means that the quality of incoming data is not good, and the final result must be unsatisfactory. The quality of data depends on the model provided by suppliers on the one hand, and on the other hand, depends on the enterprise itself. If the initial data given is wrong, no matter how good the model is, the final prediction will definitely be wrong.

The establishment of enterprise-level data warehouses has just started in China, mainly in the financial telecommunication industry with relatively complete IT infrastructure. For most enterprises, it is obviously not realistic to build a huge database of the entire enterprise from the beginning.

According to the specific situation, the headquarters can first set some normative documents to authorize and guide different branches to operate on the basis of this rule. Each branch establishes a smaller database according to different characteristics, but must communicate with each other to ensure that different databases can be integrated in the future. In this way, the establishment of an enterprise-level data warehouse is a matter of course.

Difficulty: closely combined with business goals

Data mining itself does not produce value, only the results of implementing data mining are valuable. If it is not closely connected with the core application of the business and promotes the business, it is meaningless to introduce data mining.

How can data mining be truly integrated with business goals? Enterprises need to think about a series of questions: Why should you go to data mining? What are the pressing business challenges? What is the main business problem to solve? Where does the business have room for improvement? What's the difference between not doing well now? What patterns can these gaps be addressed by discovering? What information is needed that is available?

Different industries focus on solving different business problems and have their own business pain points. For example, telecom customers lose the most, while bank customers have the most prominent credit risk. The products and solutions of different manufacturers also have their own strengths and weaknesses. When selecting models, you should start from your own needs and see which product can bring better technical support and help solve major business problems. Considering the above problems comprehensively and establishing realistic business goals, only under the guidance of this kind of thinking can ensure the maximum return on investment in data mining.

The data mining project is different from the construction of general hardware platforms. Success cannot be achieved by the IT department alone, and requires the participation of the business department. Data mining is to solve business problems, such as bank analysis of customers: which ones can bring the most profit? Which ones are at the highest risk of churn? As a result, targeted product package services, marketing and sales strategies are launched. These all come from the business sector, not the technology sector. The technical department provides support from a technical perspective and is responsible for selecting mature products and helping to control implementation risks. The close cooperation between technical departments and business departments requires the attention, intervention, planning and leadership of senior leaders.

To achieve business goals with data mining, it is also necessary to identify quantifiable metrics. It is necessary to clarify what benefits can be achieved after the system is established, and clearly define different KPIs (core performance indicators) at each stage to measure the success of the project. When a target is reached, it will motivate everyone to work towards the next stage and help advance the overall progress of the project.

In order to serve the needs of business departments, data mining should involve all internal users and design together when doing secondary application development. The system built in this way will make them feel that they can use it, the user interface is friendly, it will be used frequently, and it will work better and better. If the design is not good and it is difficult to use, users will not be willing to use it, and the purpose of promoting business will not be achieved.

Data mining is a set of overall solutions, in addition to technical factors, consulting services are also an important part. When a user is making a model selection and planning, in addition to choosing a manufacturer with mature products and leading technologies, it also depends on whether he has rich implementation experience, industry application experience, and project management experience. It is best to provide products while selling products. Business Consulting Services. This knowledge transfer of manufacturers can avoid users from taking many detours.

Emphasis: Cultivating the implementation team that truly digests technology

Zheng Xiaojun introduced the specific composition of the data mining project team: business market analysts; IT analysts; data engineering team (responsible for Extract data and prepare data. If there is a data warehouse, the data engineering team will be relatively relaxed); business users (they want to participate, depending on whether a tool is useful); project administrators and leaders.

The skills that project team members need to have are divided into four aspects: understanding data sources, understanding data preparation process, understanding algorithms, and understanding business. Among them, data preparation and understanding data sources are related, and understanding the relationship between business and algorithms is the most important.

Obviously, the members of the data mining project team come from various departments, and the requirements for personnel quality are very high. They must be able to master the corresponding technical matching and support services, in order to maximize the autonomous application of data mining in the business. Enterprises must cultivate a dedicated team that can really digest the technology, otherwise it is impossible to achieve success.

According to SAS' experience, a data mining project is a two-way interactive process. In the process of construction and implementation, on the one hand, manufacturers should be able to pass on mature product technology, rich practical experience, and practical methodology to users. On the other hand, users themselves have to digest and apply it. If it is not used, it will not bring any benefits.

Data mining is booming in foreign countries, largely due to the improvement of customer maturity. Enterprises themselves should attach importance to this to a strategic level, not just as an investment in IT projects, but also from business departments to technical departments.

Merrill Lynch has been building data warehouses for data mining since 1992. The data mining team started with only 30 people, and expanded to hundreds of people in a few years. At the same time, there are also some failure cases abroad, all because data mining is managed as a hardware investment project, ignoring the transfer and accumulation of knowledge, and ignoring the application under the guidance of methodology.