1.
Data
Mining
There
is huge amount of data available in Information Industry. This data is of no
use until converted into useful information. Analyzing this huge amount of data
and extracting useful information from it is necessary.
The
extraction of information is not the only process we need to perform; it also
involves other processes such as Data Cleaning, Data Integration, Data
Transformation, Data Mining, Pattern Evaluation and Data Presentation. Once all
these processes are over, we are now position to use this information in many
applications such as Fraud Detection, Market Analysis, Production Control,
Science Exploration etc.
Data
mining is the process of discovering actionable information from large sets of
data. Data mining uses mathematical analysis to derive patterns and trends that
exist in data. Typically, these patterns cannot be discovered by traditional
data exploration because the relationships are too complex or because there is
too much data.
Data
mining is concerned with the analysis of data and the use of software
techniques for finding hidden and unexpected patterns and relationships in sets
of data. The focus of data mining is to find the information that is hidden and
unexpected.
Data
mining can provide huge paybacks for companies who have made a significant
investment in data warehousing. Although data mining is still a relatively new
technology, it is already used in a number of industries
It is the process of collecting, searching through, and
analyzing a large amount of data in a database, as to discover patterns or
relationships
These
patterns and trends can be collected and defined as a data mining
model. Mining models can be applied to specific scenarios, such as:
a)
Forecasting: Estimating sales, predicting
server loads or server downtime
b)
Risk and probability: Choosing the best customers for
targeted mailings, determining the probable break-even point for risk
scenarios, assigning probabilities to diagnoses or other outcomes
c)
Recommendations: Determining which products are
likely to be sold together, generating recommendations
d)
Finding sequences: Analyzing customer selections
in a shopping cart, predicting next likely events
e)
Grouping: Separating customers or events into cluster of
related items, analyzing and predicting affinities
2. Data Mining Applications
Here is the list of applications of
Data Mining:
1.
Market Analysis and Management
2.
Corporate Analysis & Risk
Management
3.
Fraud Detection
4.
Other Applications
1) Market Analysis and Management
Following are the various fields of
market where data mining is used:
a)
Customer Profiling -
Data Mining helps to determine what kind of people buy what kind of products.
b)
Identifying Customer Requirements -
Data Mining helps in identifying the best products for different customers. It
uses prediction to find the factors that may attract new customers.
c)
Cross Market Analysis -
Data Mining performs Association/correlations between product sales.
d)
Target Marketing -
Data Mining helps to find clusters of model customers who share the same
characteristics such as interest, spending habits, income etc.
e)
Determining Customer purchasing pattern -
Data mining helps in determining customer purchasing pattern.
f)
Providing Summary Information -
Data Mining provide us various multidimensional summary reports
2) Corporate Analysis & Risk
Management
Following are the various fields of
Corporate Sector where data mining is used:
a)
Finance Planning and Asset Evaluation -
It involves cash flow analysis and prediction, contingent claim analysis to evaluate
assets.
b)
Resource Planning -
Resource Planning It involves summarizing and comparing the resources and
spending.
c)
Competition - It involves monitoring
competitors and market directions.
3) Fraud Detection
Data Mining is also used in fields
of credit card services and telecommunication to detect fraud. In fraud
telephone call it helps to find destination of call, duration of call, time of
day or week. It also analyze the patterns that deviate from an expected norms.
4)
Other
Applications
Data Mining also used in other
fields such as sports, astrology and Internet Web Surf-Aid.
3.
KDD
KDD
(Knowledge Discovery in Databases) is a field of computer science, which
includes the tools and theories to help humans in extracting useful and
previously unknown information (i.e. knowledge) from large collections of
digitized data. KDD consists of several steps, and Data Mining is one of them.
Data Mining is application of a specific algorithm in order to extract patterns
from data. Nonetheless, KDD and Data Mining are used interchangeably.
What is KDD?
As
mentioned above, KDD is a field of computer science, which deals with
extraction of previously unknown and interesting information from raw data. KDD
is the whole process of trying to make sense of data by developing appropriate
methods or techniques. This process deal with the mapping of low-level data
into other forms those are more compact, abstract and useful.
This
is achieved by creating short reports, modeling the process of generating data
and developing predictive models that can predict future cases. Due to the
exponential growth of data, especially in areas such as business, KDD has
become a very important process to convert this large wealth of data in to
business intelligence, as manual extraction of patterns has become seemingly
impossible in the past few decades. For example, it is currently been used for
various applications such as social network analysis, fraud detection, science,
investment, manufacturing, telecommunications, data cleaning, sports,
information retrieval and largely for marketing. KDD is usually used to answer
questions like what are the main products that might help to obtain high profit
next year in Wal-Mart.
This
process has several steps. It starts with developing an understanding of the
application domain and the goal and then creating a target dataset. This is
followed by cleaning, preprocessing, reduction and projection of data. Next
step is using Data Mining (explained below) to identify pattern. Finally,
discovered knowledge is consolidates by visualizing and/or interpreting.
What is the difference
between KDD and Data mining?
Although,
the two terms KDD and Data Mining are heavily used interchangeably, they refer
to two related yet slightly different concepts. KDD is the overall process of
extracting knowledge from data while Data Mining is a step inside the KDD
process, which deals with identifying patterns in data. In other words, Data
Mining is only the application of a specific algorithm based on the overall
goal of the KDD process.
Steps
involved in knowledge discovery process
Here is the list of steps involved
in knowledge discovery process:
a)
Data Cleaning -
In this step the noise and inconsistent data is removed.
b)
Data Integration -
In this step multiple data sources are combined.
c)
Data Selection -
In this step relevant to the analysis task are retrieved from the database.
d)
Data Transformation -
In this step data are transformed or consolidated into forms appropriate for
mining by performing summary or aggregation operations.
e)
Data Mining -
In this step intelligent methods are applied in order to extract data patterns.
f)
Pattern Evaluation -
In this step, data patterns are evaluated.
g)
Knowledge Presentation -
In this step, knowledge is represented.
4. Primary
goals of data mining
The two "high-level"
primary goals of data mining, in practice, are prediction and description.
Prediction
involves using some variables or fields in the database to predict unknown or
future values of other variables of interest.
Description
focuses on finding human-interpretable patterns describing the data.
The relative importance of
prediction and description for particular data mining applications can vary
considerably. However, in the context of KDD, description tends to be more
important than prediction. This is in contrast to pattern recognition and
machine learning applications (such as speech recognition) where prediction is
often the primary goal of the KDD process.
The goals of prediction and
description are achieved by using the following primary data mining tasks:
1)
CLASSIFICATION/PREDICTION
Classification
involves the discovery of a predictive learning function that classifies a data
item into one of several predefines classes. It involves examining the features
of a newly presented object and assigning to it a predefined class.
Classification
is a two-step process. First a model is built describing a predetermined set of
data classes or concepts and secondly, the model is used for classification.
Prediction
can be viewed as the construction and use of a model to assess the class of a
unlabeled sample, or to assess the value or value range of an attribute that a
given sample is likely to have.
According
to, any of the techniques used for classification can be adapted for use in
prediction by using training examples where the value of the variable to be
predicted is already known, along with historical data for those examples.
Typical business related questions that can be answered using classification or
prediction tasks are:
·
Which customers will buy?
·
Which products will customer by?
·
How much will customer buy?
2)
ESTIMATION
While
classification deals with discrete outcomes such as yes or no, debit card, home
loan or vehicle financing, estimation deals with continuously valued outcomes.
If some input data is available, estimation can be used to come up with some
unknown continuous variable such as income or height. In estimation, one wants
to come up with a plausible value or a range of plausible values for the
unknown parameters of a system. Typical examples of estimation and the business
related questions that can be addressed by making use it include the following:
·
How
many children are in a family?
·
Estimating
a family’s total household income.
·
Estimating
the value of a piece of property.
3)
SEGMENTATION
Segmentation
simply means making different offers to different markets segments; groups of
people defined by some combination of demographic variables such as age, gender
or income. Define the segmentation as a form of analysis used to for instance
break down the visitors to a website into unique groups with individual
behaviors.
The
grouping can then be used to make statistical projections, such as the
potential amount of purchases they are likely to make. Typical business
questions that can be answered using segmentation are:
·
What
are the different types of visitors attracted to our website?
·
In
which age groups do the listeners of a certain radio station fall into?
4)
CLUSTERING
Clustering
is the task of segmenting a diverse group into a number of similar subgroups or
clusters. Clusters of objects are formed so that objects within a cluster have
high similarity in comparison to one another, but are very dissimilar to
objects in other clusters.
Clustering
is commonly used to search for unique groupings within a data set. The
distinguishing factor between clustering and classification is that in
clustering there are no predefined classes and no examples. The objects are
grouped together based on self-similarity. Typical business question that can
be answered using clustering are:
·
What
are the groupings hidden in your data.
·
Which
customer should be grouped together for target marketing purposes?
Clustering
is grouped under descriptive data mining tasks. Clustering is best used for
finding groups of items that are similar for e.g. given a data set of
customers; identify subgroups of customers that have a similar buying behavior
5) DESCRIPTION AND VISUALIZATION
The
purpose of data mining is sometimes simply to describe what is going on in a
complicated database in a way that increased our understanding of the people,
products or processes that produced the data in the first place. They state
that a good enough description of behavior will often suggest an explanation
for it as well. One of the most powerful forms of descriptive data mining is
data visualization. Although visualization is not always easy, the right
picture can truly speak a thousand words since human beings are extremely
practiced at extracting meaning from visual scenes.
Visualization
can be useful in providing a visual representation of the location and
distribution of a company’s major clients on a map of a city or a province or
even a country
6)
Sequence/Temporal
Sequential
pattern functions analyze collections of related records and detect frequently
occurring patterns over a period of time. Difference between sequence rules and
other rules is the temporal factor
Example
- retailers database can be used to discover the set of purchases that
frequently precedes the purchase of a microwave oven
5. Steps
of Data Mining Process
In
order to systematically conduct data mining analysis, a general process is
usually followed. There is a Cross-Industry Standard Process for Data Mining
(CRISP-DM) widely used by industry members. This model consists of six phases
intended as a cyclical process
1)
Business Understanding Business understanding includes
determining business objectives, assessing the current situation, establishing
data mining goals, and developing a project plan. Tasks in this phase include:
·
Identifying
your business goals
·
Assessing
your situation
·
Defining
your data mining goals
·
Producing
your project plan
2) Data
Understanding
Once business
objectives and the project plan are established, data understanding considers
data requirements. This step can include initial data collection, data
description, data exploration, and the verification of data quality. Data
exploration such as viewing summary statistics (which includes the visual
display of categorical variables) can occur at the end of this phase. Models
such as cluster analysis can also be applied during this phase, with the intent
of identifying patterns in the data. Tasks for this phase include:
·
Gathering
data
·
Describing
·
Exploring
·
Verifying
quality
3)
Data Preparation Once
the data resources available are identified, they need to be selected, cleaned,
built into the form desired, and formatted. Data cleaning and data
transformation in preparation of data modeling needs to occur in this phase.
Data exploration at a greater depth can be applied during this phase, and
additional models utilized, again providing the opportunity to see patterns
based on business understanding. Tasks for this phase include:
·
Selecting
data
·
Cleaning
data
·
Constructing
·
Integrating
·
Formatting
4) Modeling Data mining software tools such
as visualization (plotting data and establishing relationships) and cluster
analysis (to identify which variables go well together) are useful for initial
analysis. Tools such as generalized rule induction can develop initial
association rules. Tasks for this phase include:
·
Selecting
techniques
·
Designing
tests
·
Building
models
·
Assessing
models
5)
Evaluation Model results should be
evaluated in the context of the business objectives established in the first
phase (business understanding). Gaining business understanding is an iterative procedure
in data mining, where the results of various visualization, statistical, and
artificial intelligence tools show the user new relationships that provide a
deeper understanding of organizational operations. Tasks for this phase
include:
·
Evaluating
results
·
Reviewing
the process
·
Determining
the next steps
6) Deployment Data mining can be used to both
verify previously held hypotheses, or for knowledge discovery (identification
of unexpected and useful relationships). These models need to be monitored for
changes in operating conditions, because what might be true today may not be
true a year from now. If significant changes do occur, the model should be
redone. It’s also wise to record the results of data mining projects so
documented evidence is available for future studies. Tasks for this phase
include:
·
Planning
deployment (your methods for integrating data mining discoveries into use)
·
Reporting
final results
·
Reviewing
final results
