Visualization for Clustering Methods
Editor’s note: Evie Fowler is a speaker for ODSC West. Be sure to check out her talk, “Bridging the Interpretability Gap in Customer Segmentation,” there!
At this Fall’s Open Data Science Conference, I will talk about how to bring a systematic approach to the interpretation of clustering models. To get ready for that, let’s talk about data visualization for clustering models.
Preparing a Workspace
All of these visualizations can be created with the basic tools of data manipulation (pandas and numpy) and the basics of visualization (matplotlib and seaborn).
from matplotlib import colormaps, pyplot as plt
from sklearn.cluster import KMeans
from sklearn.datasets import load_diabetes
from sklearn.preprocessing import MinMaxScaler
import numpy as np
import pandas as pd
import seaborn as snsFor this tutorial, I’ll use the diabetes prediction dataset built into matplotlib. I’ll offer a lot more insight on how to train and evaluate an effective clustering model at ODSC, but for now, I’ll just fit a few simple k-means models.
# load diabetes data
diabetesData = load_diabetes(as_frame = True).data# center and scale clusterable features
diabetesScaler = MinMaxScaler().fit(diabetesData)
diabetesDataScaled = pd.DataFrame(diabetesScaler.transform(diabetesData)
, columns = diabetesData.columns
, index = diabetesData.index)# build three small clustering models
km3 = KMeans(n_clusters = 3).fit(diabetesDataScaled)
km4 = KMeans(n_clusters = 4).fit(diabetesDataScaled)
km10 = KMeans(n_clusters = 10).fit(diabetesDataScaled)
Choosing a Color Scheme
The matplotlib package provides a number of built-in color schemes through its colormaps registry. It is convenient to choose one colormap for the entirety of a visualization, and important to choose thoughtfully. That can mean evaluating everything from whether the map is sequential (for when data can be interpreted along a scale from low to high) or divergent (for when data is most relevant at either of two extremes) to whether it is thematically appropriate for the subject (greens and browns for a topography project). When there is no particular relationship between the data and the order it will be presented in, the nipy_spectral colormap is a good choice.
# choose the nipy_spectral colormap from matplotlib
nps = colormaps['nipy_spectral']# view the whole colormap
nps
Each matplotlib colormap consists of a series of tuples, with each describing a color in RGBA format (though with components scaled to [0, 1] rather than [0, 255]). Individual colors from the map can be accessed either by integer (between 0 and 255) or by float (between 0 and 1). Numbers close to 0 correspond to colors at the lower end of the color map, while integers close to 255 and floats close to 1.0 correspond to colors at the upper end of the color map. Intuitively, the same color can be described by either an integer, or a float representing that integer as a quotient of 255.
# view select colors from the colormap
print(nps(51))
#(0.0, 0.0, 0.8667, 1.0)print(nps(0.2))
#(0.0, 0.0, 0.8667, 1.0)Creating Visualizations
Scatter Plots
The classic visualization for a clustering model is a series of scatter plots comparing each pair of features that went into the clustering model, with cluster assignment denoted by color. There are built in methods to achieve this, but a DIY approach gives more control over details like the color scheme.
def plotScatters(df, model):
""" Create scatter plots based on each pair of columns in a dataframe.
Use color to denote model label.
"""# create a figure and axes
plotRows = df.shape[1]
plotCols = df.shape[1]
fig, axes = plt.subplots(
# create one row and one column for each feature in the dataframe
plotRows, plotCols
# scale up the figure size for easy viewing
, figsize = ((plotCols * 3), (plotRows * 3))
)
# iterate through subplots to create scatter plots
pltindex = 0
for i in np.arange(0, plotRows):
for j in np.arange(0, plotCols):
pltindex += 1
# identify the current subplot
plt.subplot(plotRows, plotCols, pltindex)
plt.scatter(
# compare the i-th and j-th features of the dataframe
df.iloc[:, j], df.iloc[:, i]
# use integer cluster labels and a color map to unify color selection
, c = model.labels_, cmap = nps
# choose a small marker size to reduce overlap
, s = 1)
# label the x axis on the bottom row of sub plots
if i == df.shape[1] - 1:
plt.xlabel(df.columns[j])
# label the y axis on the first column of sub plots
if j == 0:
plt.ylabel(df.columns[i]) plt.show()
These plots do double duty, showing the relationship between a pair of features and the relationship between those features and cluster assignment.
plotScatters(diabetesDataScaled, km3)
As analysis progresses, it’s easy to focus on a smaller subset of features.
plotScatters(diabetesDataScaled.iloc[:, 2:7], km4)
Violin Plots
To get a better sense of the distribution of each feature within each cluster, we can also look at violin plots. If you’re not familiar with violin plots, think of them as the grown up cousin of the classic box plot. Where box plots identify only a few key descriptors of a distribution, violin plots are contoured to illustrate the entire probability density function.
def plotViolins(df, model, plotCols = 5):
""" Create violin plots of each feature in a dataframe
Use model labels to group.
""" # calculate number of rows needed for plot grid
plotRows = df.shape[1] // plotCols
while plotRows * plotCols < df.shape[1]:
plotRows += 1 # create a figure and axes
fig, axes = plt.subplots(plotRows, plotCols
# scale up the figure size for easy viewing
, figsize = ((plotCols * 3), (plotRows * 3))
) # identify unique cluster labels from model
uniqueLabels = sorted(np.unique(model.labels_)) # create a custom subpalette from the unique labels
# this will return
npsTemp = nps([x / max(uniqueLabels) for x in uniqueLabels]) # add integer cluster labels to input dataframe
df2 = df.assign(cluster = model.labels_) # iterate through subplots to create violin plots
pltindex = 0
for col in df.columns:
pltindex += 1
plt.subplot(plotRows, plotCols, pltindex)
sns.violinplot(
data = df2
# use cluster labels as x grouper
, x = 'cluster'
# use current feature as y values
, y = col
# use cluster labels and custom palette to unify color selection
, hue = model.labels_
, palette = npsTemp
).legend_.remove()
# label y axis with feature name
plt.ylabel(col) plt.show()plotViolins(diabetesDataScaled, km3, plotCols = 5)
Histograms
Violin plots show the distribution of each feature within each cluster, but it is also helpful to look at how each cluster is represented in the broader distribution of each feature. A modified histogram can illustrate this well.
def histogramByCluster(df, labels, plotCols = 5, nbins = 30, legend = False, vlines = False):
""" Create a histogram of each feature.
Use model labels to color code.
"""
# calculate number of rows needed for plot grid
plotRows = df.shape[1] // plotCols
while plotRows * plotCols < df.shape[1]:
plotRows += 1# identify unique cluster labels
uniqueLabels = sorted(np.unique(labels))
# create a figure and axes
fig, axes = plt.subplots(plotRows, plotCols
# scale up the figure size for easy viewing
, figsize = ((plotCols * 3), (plotRows * 3))
)
pltindex = 0
# loop through features in input data
for col in df.columns:
# discretize the feature into specified number of bins
tempBins = np.trunc(nbins * df[col]) / nbins
# cross the discretized feature with cluster labels
tempComb = pd.crosstab(tempBins, labels)
# create an index in the same size as the cross tab
# this will help with alignment
ind = np.arange(tempComb.shape[0]) # identify the relevant subplot
pltindex += 1
plt.subplot(plotRows, plotCols, pltindex)
# create grouped histogram data
histPrep = {}
# work one cluster at a time
for lbl in uniqueLabels:
histPrep.update(
{
# associate the cluster label...
lbl:
# ... with a bar chart
plt.bar(
# use the feature-specific index to set x locations
x = ind
# use the counts associated with this cluster as bar height
, height = tempComb[lbl]
# stack this bar on top of previous cluster bars
, bottom = tempComb[[x for x in uniqueLabels if x < lbl]].sum(axis = 1)
# eliminate gaps between bars
, width = 1
, color = nps(lbl / max(uniqueLabels))
)
}
)
# use feature name to label x axis of each plot
plt.xlabel(col)
# label the y axis of plots in the first column
if pltindex % plotCols == 1:
plt.ylabel('Frequency')
plt.xticks(ind[0::5], np.round(tempComb.index[0::5], 2))
# if desired, overlay vertical lines
if vlines:
for vline in vlines:
plt.axvline(x = vline * ind[-1], lw = 0.5, color = 'red')
if legend:
leg1 = []; leg2 = []
for key in histPrep:
leg1 += [histPrep[key]]
leg2 += [str(key)]
plt.legend(leg1, leg2) plt.show()
histogramByCluster(diabetesDataScaled, km4.labels_)
This process scales easily when more cluster categories are needed.
histogramByCluster(diabetesDataScaled, km10.labels_)
Conclusion
These visualizations will provide a strong base for evaluating clustering models. For more about how to do so in a systematic way, be sure to come to my talk at this Fall’s Open Data Science Conference in San Francisco!
About the Author:
Evie Fowler is a data scientist based in Pittsburgh, Pennsylvania. She currently works in the healthcare sector leading a team of data scientists who develop predictive models centered on the patient care experience. She holds a particular interest in the ethical application of predictive analytics and in exploring how qualitative methods can inform data science work. She holds an undergraduate degree from Brown University and a master’s degree from Carnegie Mellon.
Originally posted on OpenDataScience.com
Read more data science articles on OpenDataScience.com, including tutorials and guides from beginner to advanced levels! Subscribe to our weekly newsletter here and receive the latest news every Thursday. You can also get data science training on-demand wherever you are with our Ai+ Training platform. Interested in attending an ODSC event? Learn more about our upcoming events here.
