Focusing on the Data in Data Mining: Lessons from Recent Experience

The use of data mining is growing rapidly. The number of data mining consultants, as well as the number of commercial tools available to the "non-expert" user, are also quickly increasing. It is becoming easier than ever to collect datasets and apply data mining tools to them. As more and more non-experts seek to exploit this technology to help with their business, it becomes increasingly important that they understand the underlying assumptions and biases of these tools. There are a number of factors to consider before applying data mining to a database. In particular, there are important issues regarding the data which should be examined before proceeding with the data mining process. While these issues may be well-known to the data mining expert, the non- expert is often unaware of their importance. In this paper, we will focus on three specific issues, and illustrate each through the use of examples taken from our recent experiences. For each issue, we provide insight into how it might be problematic and suggest techniques for approaching such situations.