AMME4710 Computer Vision and Image Processing
Tutorial Activity Week 6, 2023
Image Segmentation
• This tutorial activity is to be completed during the Week 6 tutorial (Friday 8th September
2023)
• When you have completed the activity, you will need to show your results to the tutor who
will check your work and check that you have sufficiently completed the task
Objectives
• This tutorial will introduce you to concepts in image segmentation and explore methods for
semi-supervised segmentation and unsupervised image segmentation routines based on image
clustering
1. Active Contours and Snakes
“Active contours” (or snakes) is a semi-automated image segmentation technique that attempts to
refine an initial contour to best fit a target shape in an image. The algorithm attempts to find an
optimal position for a closed curve which best fits to the boundary of an object in the image, based
on an energy minimisation routine which is a combination of fit to the edge while maintaining the
shape of the contour via terms controlling the elasticity and curvature of the contour.
Download the file “example_code_week6.zip” from Canvas and open up the files “snakes.m” and
“snakes_demo.m”. This code implements a variation on the snake algorithm known as a “balloon-
force snake”. This approach is a modification to the objective function presented in the lecture that
also applies a “pressure force” term to the energy function that makes the curve behave like a balloon
[1]. In this model, the contour moves out (inflation) or in (deflation) along its normal vectors thus
eliminating the need for the initial curve to be close to the solution (correct edges) to converge. The
Balloon Snake passes over relatively weak edges and it stops on salient edges. By placing balloon
snake inside/outside the object of interest and inflate/deflate contour to its boundary, this new model
is useful for objects that are difficult to locate or too complex to trace by a traditional snake
algorithm.
Run the demonstration MATLAB script “snakes_demo.m”. The demonstration presents the user with
an image and an option to click points to form an initial contour around the object. The first image
shown is an MRI image of a heart cavity. Click points along the inside of the small dark region in the
center of the image. Make sure you click points in a clockwise direction. For the second image
(pelican in flight) click points along the outside of the pelican. After you have clicked points, a
snakes algorithm is run to refine the initial curve to fit each target shape using the function snake.
The function snake (in “snake.m”) takes as an input an image, calculated gradient values, an initial
curve (represented by a series of 2D points) and four parameters that control the energy function:
• alpha: controls the elasticity term of the curve (internal energy)
• beta: controls the rigidity/curvature term of the curve (internal energy)
• kappa: controls the external energy term associated with the image gradient
• lambda: controls the balloon force term: this term is a constant force that makes the contour
continue to expand (positive values) or contract (negative values) until the contour is reached.
Tutorial Activity 1: Snakes
• Have a read through the demo program “snakes_demo.m” and get an understanding of how
the two examples are performed
• Make a copy of this script and try running the algorithm with different input images and
varying the parameters alpha, beta, kappa and lambda. Note that snake expects gradients
that are computed over a grayscale image: make sure you convert images to grayscale if
required.
• You may need to vary the filter parameters for which the gradient is calculated over, and for
colour images you may need to apply a colour transformation to accentuate the gradient
around the object you are targeting. See the second half of the example to see how a weighted
combination of image channels can be used to maximise the difference along a target object
contour.
2. Image Segmentation using K-means clustering
K-means is a data clustering algorithm designed to split N data points into K clusters such that the
within cluster sum of squares is minimised. K-means can be used to segment the pixels in an image
into K distinct classes based on colour, intensity, textural, spatial and/or other properties of the pixels
themselves in an automated way.
MATLAB provides an implementation of k-means clustering via the function kmeans. kmeans
takes an N x M matrix of input data points (N points, each with M dimensions) and a scalar K and
returns a list of cluster index values for each point, and the K cluster centers. To apply kmeans to
cluster the pixel values in a colour image, you first need to convert the image data into a list of
points. For example, to cluster based on RGB colour alone into K = 4 clusters:
points = reshape(im, size(im,1)*size(im,2), 3);
[cluster_idx, cluster_center] = kmeans(points, 4);
where im is a three-channel colour image (RGB). To transform the resulting list of indices into a
segmented image, the list must be reshaped into an image array:
pixel_labels = reshape(cluster_idx, size(im,1), size(im,2));
imagesc(pixel_labels)
The function kmeans provides a variety of optional parameters that can control the type of distance
metric used to compute distance between points (defaults to a standard Euclidean distance) and the
number of replicate the clustering process to avoid unlucky convergence (you should set the
parameter 'Replicates' to a number greater than 1). To visualise a clustered pointcloud in a (up
to) 3D feature space (i.e. clusters based on RGB), you can use the function scatter3, which
produces a 3D scatter plot and provides the ability to colour the points. For example:
[m,n,d] = size(im);
scatter3(points(:,1),points(:,2),points(:,3),ones(n*m),1),c)
colours the points using the vector c, which can be an n*m by 1 vector specifying the cluster number
of each point (i.e. from cluster_idx) or an n*m by 3 vector of per-point colours (RGB values
between 0 and 1). For example, to colour the points based on cluster id, with an equal spread of hues
for each cluster:
c = zeros(n*m,3);
for i = 1:length(cluster_idx)
c(i,:) = hsv2rgb([cluster_idx(i)/K,1,1]);
end
Produces a vector of colours where each point is coloured by its cluster index, where clusters are
coloured by equally spreading out values in hue.
Tutorial Activity 2: K-means clustering and Image Segmentation
• Develop MATLAB code to perform image segmentation based on k-means colour clustering
for a user-selectable value of K
• Experiment with the performance of this segmentation approach in the example images by
varying K. Try varying the input colour space of the image array (i.e. by using the functions
rgb2hsv, rgb2lab etc.) prior to clustering and note the way in which data is clustered
differently in difference colour spaces
• Extend you approach to include either a texture response layer and/or a spatial layer:
• For adding spatial information into the feature space, add columns to the data matrix in points
that represent the x and y locations of each pixel in the data point list, and include these in the
clustering.
• For adding a textural layer, implement a simple texture filter (see week 3 tutorial) and add the
output of this filter as an additional column in the data matrix.
• Note that the range of values (in feature space) of the spatial and textural dimensions are
different to the 0-255 colour space values provided by the RGB dimensions. You will need to
scale your spatial/textural values to control the degree to which clusters lock onto
colour/spatial/textural properties (i.e. if these dimensions have a range that is greater than 0-
255, more emphasis will be put on segmenting these dimensions, and having numbers that
have a smaller range than 0-255 will put less emphasis with respect to colour.
3. Interactive Image Segmentation using Lazy Snapping
“Lazy Snapping” [3] is an interactive segmentation method that uses a collection of “seed” image
locations corresponding to foreground or background objects and uses these seeds in a graph energy-
based optimisation algorithm to segment a foreground object from a background one. The approach
enables a user to incrementally improve a segmentation result from an initial segmentation by
progressively adding foreground and background labels.
MATLAB provides a function “lazysnapping” [3] which can be used to perform this
segmentation. The function takes in four arguments: the image data to be segmented, and label
matrix L, a foreground mask image and a background mask image. The function operates on a
region-by-region level (rather than pixel-by-pixel) using the superpixel regions provided by the label
matrix L.
To provide superpixel regions to lazysnapping, MATLAB provides a function “superpixels”
which is an implementation of the SLIC super-pixel algorithm [4]. This function outputs a label
matrix which indicates the pixel-wise membership in each identified cluster/superpixel. To visualise
segmented superpixels, you can use the function “boundarymask” to create a Boolean image that
highlights the boundary pixels between superpixels, for example:
im = imread('example_images_week6/apples.jpg');
[L,N] = superpixels(im,500);
figure
BW = boundarymask(L);
imshow(imoverlay(im,BW,'cyan'))
Once you have a label matrix L of superpixels, you can also use the function “label2idx” to
produce a cell array where each element contains a list of image index values for the each one of the
superpixels: this can be used to read image data or assign image data for each superpixel.
To begin exploring how these functions work, we will use MATLAB’s “Image Segmenter” tool,
which you can find by clicking on the apps tab, and looking under “Image Processing and Computer
Vision”. Once the app window has loaded, click on the “Load Image” button on the top left to load
in an image to start working with. Once the image has loaded, click on the “Graph Cut” tool to
perform interactive segmentation using the Lazy Snapping algorithm.
Start by clicking on the “Mark Foreground” button on the top right and use the cursor to mark a
selection of pixels corresponding to some foreground regions. Once complete, click on “Mark
Background” and mark some background regions. Once you have some foreground and background
marking available, the app will automatically apply a Lazy Snapping foreground/background
segmentation algorithm and display the result to screen. You can continue to label additional
foreground and background pixels to adjust the clusters used to apply the segmentation using a graph
energy-based optimisation.
Once you have completed your labels, you can click “Create Mask” and use the app to export the
resulting binary mark that splits the foreground/background pixels. You can also click “Export”,
“Generate Function” to see the MATLAB code used (including labelled pixel locations/indices for
foreground/background) to generate the segmentation. You could potentially overlay you segmented
foreground object on any other background image (a bit like a Zoom background):
Tutorial Activity 3: Interactive Image Segmentation
• Load up some of the images found in this week’s examples and use the Image Segmenter tool
to generate a mask for some of the foreground objects from each image’s background.
• Export code from the Image Segmenter App and modify this render your segmented
foreground object onto a second background image of your choice: you might like to use a
webcam image of yourself to create a faux-Zoom backdrop
• OPTIONAL EXTENSION: You could combine this code in a loop while reading each
frame of your webcam (see code in Week 3 Tutorial for capturing webcam frames) and
display the segmented foreground object from each camera frame (like a Zoom backdrop).
Your foreground/background regions will need to be designed to sit in regions that would
always be background or foreground in each frame (i.e. center/bottom of image for
foreground etc.) assuming the person remains roughly in the center of the image.
References and Further Reading:
[1] Cohen, L., Cohen, I., 1993. Finite element methods for active contour models and balloons
for 2D and 3D images. IEEE Trans. on Pattern Analysis and Machine Intelligence 15(11),
1131–1147, 1993.
[2] J. Liang, T. McInerney, D. Terzopoulos, “United Snakes”, Medical Image Analysis, 10, 215–
233, 2006.
[3] Y. Li, J. Sun, C. Tang and H. Shum, “Lazy Snapping”, ACM Transactions on Graphics,
2004.
[4] R. Achanta et. al., “SLIC superpixels compared to state-of-the-art superpixel methods”, IEEE
Trans. on Pattern Analysis and Machine Intelligence, 2012.
[5] R. Szeliski, “Computer Vision: Algorithms and Applications”, Springer, 2010 (Section 5)
[6] MATLAB Image Processing Toolbox Documentation,
https://au.mathworks.com/products/image.html, Mathworks, 2023.