Notebook 9 – k-means-checkpoint
$k$-means
$k$-means is one of the most commonly used clustering algorithms that clusters the data points into a predefined number of clusters. The following parameters are available:
k is the number of desired clusters.
maxIterations is the maximum number of iterations to run.
initializationMode specifies either random initialization or initialization via $k$-means.
runs is the number of times to run the $k$-means algorithm ($k$-means is not guaranteed to find a globally optimal solution, and when run multiple times on a given dataset, the algorithm returns the best clustering result).
initializationSteps determines the number of steps in the $k$-means algorithm.
epsilon determines the distance threshold within which we consider $k$-means to have converged.
initialModel is an optional set of cluster centers used for initialization. If this parameter is supplied, only one run is performed.
First, the usual initializations with the relevant imports.
In [ ]:
import org.apache.spark.sql.SparkSession
import org.apache.spark.mllib.util.MLUtils
import org.apache.spark.mllib.clustering.{KMeans, KMeansModel}
import org.apache.spark.mllib.linalg.Vectors
import org.apache.spark.mllib.linalg.Matrix
import org.apache.spark.mllib.linalg.Vectors
import org.apache.spark.mllib.linalg.distributed.RowMatrix
In the cell below, we will start our small scale $k$-means clustering experiment. We will use the Spambase dataset within the Jupyter notebook (you can download the dataset from the link given, but for the notebook, it’s available at files/spambase.data).
First step involves loading and parsing the data. Note that the data has known format, and it includes a label, classifying each instance as spam or not spam in the final column. This will need to be taken off for the purposes of clustering.
In [ ]:
val sparkSession = SparkSession
.builder()
.master(“local[1]”)
.appName(“k-means”)
.getOrCreate()
val sc = sparkSession.sparkContext
// Load the data
val data = sc.textFile(“files/spambase.data”)
// Split off the labels and parse the remaining data (known format)
val labels = data.map(s => Vectors.dense(s.split(‘,’).takeRight(1).map(_.toDouble)))
val parsedData = data.map(s => Vectors.dense(s.split(‘,’).take(57).map(_.toDouble))).cache()
Now we use the KMeans object to cluster the data into two clusters. We pass the number of clusters to the algorithm. To evaluate the clustering, we compute the Within Set Sum of Squared Error (WSSSE). This can be reduced by increasing $k$.
In [ ]:
// Cluster the data into two classes using KMeans
val numClusters = 2
val numIterations = 200
val clusters = KMeans.train(parsedData, numClusters, numIterations)
// Evaluate clustering by computing Within Set Sum of Squared Errors
val WSSSE = clusters.computeCost(parsedData)
println(“Within Set Sum of Squared Errors = ” + WSSSE)
// Shows the result.
println(“Cluster Centers: “)
clusters.clusterCenters.foreach(println)
The MLlib implementation includes a parallelized variant of the k-means++ method called k-means||. Both these techniques are ways to initialize the centroids rather than picking at random. In k-means++, the decision regarding each centroid is dependent on the previous decision, so it cannot be parallelised. k-means|| is parallelizable.
Exercises
Exercise 1
Make the $k$-means implementation a standalone program. You should run your HPC application on the KDDCUP1999 data with 4M points. The dataset is available here.
Exercise 2
The details of k-means|| are available in the paper linked above. The algorithm details are reproduced for you here:
For this week’s exercise, you should implement the centroid initialization directly. Some of the functions you will need to implement are indicated below.
In [ ]:
// Sampling a point with a probability
// Calculating / updating phi_X(C)
// Main algorithm