DATA3888: Data Science Capstone
Week 2: Case Study 3 - Brainbox
Andy Tran
26 February, 2023
2 / 28
Outline
Background + data
Research Task
Evaluation Strategies
3 / 28
Background and data
Case study: Brainbox data
5 / 28
Case study: Brainbox data
In the brain box exercise, we aim to identify the left or the right eye movement. We will use the function
readWave in tuneR to read in the .wav le.
library(tuneR)
library(ggplot2)
wave_LRL <- readWave("LRL_L1.wav")
time_LRL <- seq_len(length(wave_LRL))/wave_LRL@samp.rate # time (in second) of the sequencing
wave_LRL
##
## Wave Object
## Number of Samples: 88246
## Duration (seconds): 8.82
## Samplingrate (Hertz): 10000
## Channels (Mono/Stereo): Mono
## PCM (integer format): TRUE
## Bit (8/16/24/32/64): 16
6 / 28
Explanations of the slotNames
slotNames(wave_LRL)
## [1] "left" "right" "stereo" "samp.rate" "bit" "pcm"
"left": Recordings (as sample point) in the left channel.
"right": Recordings (as sample point) in the right channel.
"stereo": Whether the sounds are recorded in stereo.
"samp.rate": The number of samples per second. If the sampling rate is 4000 hertz, a recording with
a duration of 5 seconds will contain 20,000 samples.
"bit": the number of bits in each sample, and can be considered as a form of resolution.
"pcm": stands for Pulse Code Modulation, is a method to capture audio waveforms digitally.
7 / 28
Visualise data
We can use the base R function to visualise the data:
plot(time_LRL, wave_LRL@left, type = "l", ylim = c(-3000, 3000))
Can you determine the event time and event type by eye?
8 / 28
Visualise data
We can use the base R function to visualise the data:
plot(time_LRL, wave_LRL@left, type = "l", ylim = c(-3000, 3000))
We can determine the event time and event type by eye.
eventTime_LRL <- c(2, 4, 6)
eventType_LRL <- c("L", "R", "L")
eventTable_LRL <- data.frame(eventTime_LRL, eventType_LRL) 9 / 28
Identify eye-movement
ggplot(df, aes(x = time, y = Y, col = class, group = 1)) + geom_line() +
scale_color_manual(values = c("#E41A1C", "#377EB8", "black")) +
theme_bw()
10 / 28
Research Task
We aim to predict whether the eyeball moves to
the left or the right with the input signals. This
problem requires prediction in real-time at
multiple levels.
1. Identify when an event starts or ends (eye
movement).
2. Identify whether the eyeballs to the left or the
right.
What makes this different to our usual
classication problems?
How would you approach this?
Illustration: a 0.5 second moving window.
Research Task
12 / 28
How to approach this problem?
We can turn this into a two-part classication problem.
In each window, we can:
1. Classify into "event" or "non-event"
2. If it is an "event", classify into "Left" or "Right"
Extra considerations:
We will need to perform this classication in real time.
How would we evaluate a model?
13 / 28
Training data
For a length of 88246 measurements in 8.8 seconds, we have 83247 windows.
windowSize = 0.5; xtime = time_LRL; Y = wave_LRL@left
ind = seq_len(which(xtime == xtime[length(xtime)] - windowSize) + 1); length(ind)
## [1] 83247
Downsampling
To reduce the computational time.
Select 1 out of every 50 windows and calculated statistics within each window.
# downsample for speed:
downSampleRate = 50
ind = seq(1, length(ind), by = downSampleRate); length(ind)
## [1] 1665
14 / 28
What is a possible classication rule?
For example, classify based off the standard
deviation of the signal. Do you have other
ideas?
Step 1: Identify eye-movement
testStat = rep(NA, length(ind))
stat_event <- rep(NA, length(ind))
for (i in 1:length(ind)) {
# Calculate the standard deviation of signals within the window
testStat[i] <- sd(Y[xtime >= xtime[ind[i]] & xtime < xtime[ind[i]] + windowSize])
# See if the window belongs to which event type
event_table <- table(df$class[xtime >= xtime[ind[i]] & xtime < xtime[ind[i]] + windowSize])
stat_event[i] <- names(event_table)[which.max(event_table)]
}
15 / 28
Step 1: Identify eye-movement
Event = Either left or right eye-movements.
Event: {SD(signal) > 250}
No Event: {SD(signal) <= 250}
thresholdEvents <- 250
predictedEvent <- which(testStat > thresholdEvents)
timeMiddle <- xtime[ind] + windowSize/2 # the middle of the time of a window
eventTimes <- timeMiddle[predictedEvent]
16 / 28
predictedEvent[1:10]
## [1] 232 233 234 235 236 237 238 239 240 241
stripchart(predictedEvent, xlab = "predicted event wind
How many events?
17 / 28
How to combine events?
One way is to look for the big gap of the indices and then identify the end points of the interval.
deltaSameEvent <- windowSize
gaps <- which(diff(eventTimes) > deltaSameEvent)
event_time_interval <- min(eventTimes)
for (i in 1:length(gaps)) {
event_time_interval <- append(event_time_interval,
c(eventTimes[gaps[i]],
eventTimes[gaps[i] + 1]))
}
event_time_interval <- append(event_time_interval, max(eventTimes))
event_time_interval <- matrix(event_time_interval, ncol = 2, byrow = TRUE)
event_time_interval
## [,1] [,2]
## [1,] 1.4051 2.4151
## [2,] 3.5701 4.4451
## [3,] 5.5051 6.4801
Can you think of another approach?
18 / 28
The points within these intervals can be considered as signals representing eye movement.
df$predictedEventTimes <- FALSE
for (i in 1:nrow(event_time_interval)) {
df$predictedEventTimes[event_time_interval[i, 1] <=
df$time & event_time_interval[i, 2] >= df$time] = TRUE
}
ggplot(df, aes(x = time, y = Y, col = predictedEventTimes, group = 1)) +
geom_line() + scale_color_manual(values = c("black", "#E41A1C")) + theme_bw() +
geom_vline(xintercept = as.vector(event_time_interval), lwd = 1, lty = 2, col = "purple")
19 / 28
Step 2: Identify left or right
20 / 28
Step 2: Identify left or right
If we remove non-events, what do we get?
21 / 28
Step 2: Identify left or right
Section 2 in the Week 3 computer lab introduces the function tsfeatures (from the tsfeatures
package) which summarises a time series data set into 36 numerical features.
This allows us to convert our wave data into a typical matrix format, where we can apply our
standard classication tools!
22 / 28
Summary
We can use a simple decision rule to classify data into eye-movements and non eye-movement.
If we have an event, we summarise it into some numerical features using tsfeatures , then perform
a standard classication into Left or Right.
But for a given classier, how would we assess its performance? What evaluation metrics would we
use?
23 / 28
Evaluation Strategies
Evaluation strategies
In practice, we are using ML at different levels
Classify eye-movements and non eye-movement.
Classify Left vs Right movement.
Both parts are essential components. How challenging is each problem?
In deployment, the output will be a sequence of L and R movements. That is, we need to accurately
determine the full sequence and not just individual L and R. This is a different problem.
Eg. if the true signal is "LLR" and the prediction is
"LLLR"
"LRR"
"LLRR"
Should we consider all 3 as inaccurate or is any prediction more acceptable? How will we measure
that?
25 / 28
Evaluation metrics - string metrics
A string metric (or string similarity/distance metric) refers to a metric that measures distance
between two text strings or two sequences.
The most widely used string metric is the Levenshtein distance named after the Soviet
mathematician Vladimir Levenshtein.
Informally, the Levenshtein distance between two words is the "minimum number of single-character
changes that is required to turn one word into the other". Single-character modications may be
insertions, deletions or substitutions.
Other metrics includes Hamming distance, Simple matching coecient (SMC), Jaccard coecient,
Maximal matches and many others.
26 / 28
Evaluation metrics - string metrics
R package stringdist provide a number of implementations
library(stringdist) ## help(stringdist)
stringdist("LLR","LLR", method="lv")
## [1] 0
stringdist("LLR","LLRR", method="lv")
## [1] 1
stringdist("LLR","LRR", method="lv")
## [1] 1
stringdist("LLRLRRRR","RRLRLR", method="lv")
## [1] 4
27 / 28
Interdisciplinary project - design
The DATA3888 and PHYS3888 tutors have designed and generated a training data set. This data set
tries to capture a few variables:
Individual effect: use two tutors
Length of the streaming: create WAV le with different length of signals.
What other training data would you generate?
Could we extend our framework to capture other actions (not just L or R)?