Sarah Juarez

Linear Regression Model

2019-10-03T00:00:00+00:00

Predicting Sound Pressure in Wind Tunnel Using Linear Regression Model:

The main purpose of this project was to predict the sound pressure in an anechoic wind tunnel. This was done using data from the Airfoil Self-Noise Data Set in the following UCI Machine Learning Repository: link The rows of these files represent the data samples, while columns 1-5 are the features and the sound pressure output is column 6, as described below:

Frequency, in Hertz
Angle of attack, in degrees
Chord length, in meters
Free-stream velocity, in meters per second
Suction side displacement thickness, in meters
Scaled sound pressure level, in decibels

Data exploration: To begin, I first plotted a histogram of each feature and the output to see how they are distributed. What I found was that feature 1 is distributed mostly around 0-5000 Hertz, feature 2 is a little more evenly distributed, with the highestest distributions around 0-1 degrees and 10 degrees. Features 3 and 4 are distributed in 4 specific values, the rest of the values having no samples and feature 5 has the most samples between 0 and 0.01 meters. The outcome distribution increases until it peaks at around 128 decibels and then it decreases.

# read in training data
training_data = read_csv("airfoil_self_noise_train.csv")

# plot the histogram of each feature and outcome
fig, ax = plt.subplots(2,3)
fig.set_size_inches(9.25, 5.5)

ind = 0 
for i in range(2):
    for j in range(3):
        f1 = training_data.iloc[:,ind]
        ax[i][j].hist(f1)
        if (ind + 1) == 6:
            ax[i][j].set_title("Outcome")
        else:
            ax[i][j].set_title("Feature " + str(ind + 1))
        ind = ind + 1
plt.tight_layout()

Implementing a Linear Regression Model: After that, I implemented a linear regression model using the ordinary least squares (OLS) solution. In order to use this OLS formula, $w^{∗} = (X^{T} X)^{−1}X^{T} y$ , I built a data matrix whose rows corresponded to the training samples and whose columns corresponded to the features (including the constant 1 for the intercept). I also defined a function to calculate the OLS using the formula as reference.

# function for ordinary least squares (OLS)
# 1) get transpose of X
# 2) calculate first half of equation: the inverse of the 
#    dot product of X transpose and X
# 3) calculate the second half of the equation: the dot 
#    product of X transpose and y

def OLS(X, y):
    transpose_X = X.transpose()
    first_half = np.linalg.inv(np.dot(transpose_X, X))
    second_half = np.dot(transpose_X, y)
    return np.dot(first_half, second_half)

# create dataframe for outcome values 
# y
outcomes = DataFrame(training_data.iloc[:,5])

# add column of 1's and remove the outcome column
# X
training_data.insert(0, "constant", [1] * len(training_data.iloc[:,1]), True)
training_data.drop('outcome', axis=1, inplace=True)

# use OLS to get weights
calculated_weights = OLS(training_data, outcomes)
calculated_weights

array([[ 1.32303685e+02],
       [-7.11322380e-04],
       [-4.54774159e-01],
       [-5.23017057e+01],
       [ 8.39826774e-02],
       [-9.14287813e+01]])

The liner regression model using the ordinary least squares solution has 6 parameters in this case.

After calculating the OLS solution, I tested the model on the test data and computed the residual sum of squares error (RSS) between the actual and predicted outcome variable. I ended up getting an RSS value of about 72.

# function to calculate the residual sum or squares error (RSS) 
# between the actual and predicted outcome
# 1) calculate the dot product of X and weights
# 2) calculate first half of equation: the transpose of the actual 
#    outcomes minus the dot product of X and the weights
# 3) calculate second half of equation: the actual outcomes minus 
#    the dot product of X and the weights
# 4) get the square root of the dot product of the first and second 
#    half of the equation

def RSS(X, w, y):
    x_dot_w = np.dot(X, w)
    first = (y - x_dot_w).transpose()
    second = y - x_dot_w
    return sqrt(np.dot(first, second))

# read in testing data
test_data = read_csv("airfoil_self_noise_test.csv")

# create dataframe for outcome values 
# y
test_outcomes = DataFrame(test_data.iloc[:,5])

# add column of 1's and remove the outcome column
# X
test_data.insert(0, "constant", [1] * len(test_data.iloc[:,1]), True)
test_data.drop('outcome', axis=1, inplace=True)

# calculate RSS
rss = RSS(test_data, calculated_weights, test_outcomes)
rss

71.66865682786033

We could stop there, but I decided to experiment with a few different feature combinations instead of using all of them to see if I could get a lower RSS value. You can see the results of this below, Using features 3 and 4 I got an RSS of about 133, using features 1, 2, and 5 I got an RSS of about 107 and when using features 1 and 4 I got an RSS of about 85.

# using features 3 and 4 
# remove other features from training data
training_data_f34 = training_data.drop(['f1', 'f2', 'f5'], axis=1, inplace=False)

# OLS for weights
calculated_weights_f34 = OLS(training_data_f34, outcomes)
print("weights:\n", calculated_weights_f34)

# remove other features from test data
test_data_f34 = test_data.drop(['f1','f2','f5'], axis=1, inplace=False)

# calculate RSS
rss = RSS(test_data_f34, calculated_weights_f34, test_outcomes)
print("\nRSS:", rss)

weights:
 [[ 1.25761164e+02]
 [-5.44150127e+01]
 [ 6.06585582e-02]]

RSS: 133.45352324493342

# using features 1, 2 and 5
# remove other features from training data
training_data_f125 = training_data.drop(['f3', 'f4'], axis=1, inplace=False)

# OLS for weights
calculated_weights_f125 = OLS(training_data_f125, outcomes)
print("weights:\n", calculated_weights_f125)

# remove other features from test data
test_data_f125 = test_data.drop(['f3','f4'], axis=1, inplace=False)

# calculate RSS
rss = RSS(test_data_f125, calculated_weights_f125, test_outcomes)
print("\nRSS:", rss)

weights:
 [[ 1.32516436e+02]
 [-5.76514811e-04]
 [-1.46030326e-01]
 [-2.60904487e+02]]

RSS: 107.44085921265567

# using features 1 and 4
# remove other features from training data
training_data_f14 = training_data.drop(['f2', 'f3', 'f5'], axis=1, inplace=False)

# OLS for weights
calculated_weights_f14 = OLS(training_data_f14, outcomes)
print("weights:\n", calculated_weights_f14)

# remove other features from test data
test_data_f14 = test_data.drop(['f2', 'f3', 'f5'], axis=1, inplace=False)

# calculate RSS
rss = RSS(test_data_f14, calculated_weights_f14, test_outcomes)
print("\nRSS:", rss)

weights:
 [[ 1.23150041e+02]
 [-1.39844979e-04]
 [ 5.75414946e-02]]

RSS: 85.24503305195404

While I wasn’t able to get and RSS value smaller than when I used all the features, I did not test all possible combinations, therefore, it might be possible to achieve a smaller RSS value with a different combination of features. The best way to go about this would probably be to write a program that goes through every possible feature combination and outputs the one with the lowest RSS value, the result from that would be the feature combination that would be the most useful to use when making predictions.

Linear Perceptron Model

2019-09-25T00:00:00+00:00

Linear Perceptron Algorithm

The purpose of this project was to break down the linear perceptron algorithm step-by-step to demonstrate how it works. In order to do this, I set up three training samples in a 2D space. These samples are:

Sample $x_{1}$ with coordinates (1, 3) belonging to Class 1 ( $y_{1}$ = 1)
Sample $x_{2}$ with coordinates (3, 2) belonging to Class 2 ( $y_{2}$ = −1)
Sample $x_{3}$ with coordinates (4, 1) belonging to Class 2 ( $y_{3}$ = −1)

For this project, the linear perceptron is initialized with a line with corresponding weight $w(0) = [2, -1, 1]^{T}$ (this can also be written as the line $2 − x + y = 0$ ).

To start off, I simply plotted $x_{1}$ , $x_{2}$ , and $x_{3}$ , the starting weight line w(0) and the direction of the weight w(0) on the line. I also defined two functions, one for plotting these three things and another for computing dot product for the next part of the project.

# function for plotting the sample points, weight line 
# and the direction of the weight

def plot_graph(x, y, sample_names, colors, x0, y0, u, v, label_name):
    # plot points
    fig, ax = plt.subplots()
    ax.scatter(x, y, c=colors)
    for i, txt in enumerate(sample_names):
        ax.annotate(txt, (x[i] * (1.03), y[i]* (1.03)))
    
    # plot weight line
    plt.plot(x0, y0, '-k', label=label_name)
    plt.legend()
    
    # plot weight direction
    plt.quiver(x0[25], y0[25], u, v, scale=5)
    ax.set_aspect(1./ax.get_data_ratio())
    

# function for computing dot product and printing results
def inner_product(line, samples, line_name):
    for i in np.arange(3):
        print("The inner product of " + line_name + " and x" + str(i+1) + ":", np.dot(line, samples[i]))

# data for samples x1, x2, x3 
x = [1, 3, 4]
y = [3, 2, 1]

sample_names = ['x1', 'x2', 'x3']
colors = ['red' if i == 1 else 'blue' for i in x]

# data for line corresponding to weight w(0)
x0 = np.linspace(0,5)
y0 = x0-1

# plot data points and weight line w(0) with direction
plot_graph(x, y, sample_names, colors, x0, y0, -1, 1, 'w(0)')

For the next step, I used the rule $sign(w(t)^{T}x_{n})$ , to determine which samples were correctly and incorrectly classified using the weight w(0). To do this, I first wrote the data points and weight as vectors and then computed the inner product between the weight and each sample to see which samples were misclassified (based on the inner product being greater or less than zero).

# data points as vectors
x1 = np.array([1, 1, 3])
x2 = np.array([1, 3, 2])
x3 = np.array([1, 4, 1])
samples = [x1, x2, x3]

# weight line as vector
w0 = np.array([2, -1, 1])

# inner product of weight line w(0) and data samples
inner_product(w0, samples, 'w0')

The inner product of w0 and x1: 4
The inner product of w0 and x2: 1
The inner product of w0 and x3: -1

In this case, the inner product of w(0) and $x_{1}$ should be greater than zero because $x_{1}$ is part of class 1 and the inner products of w(0) and $x_{2}$ as well as w(0) and $x_{3}$ should both be less than zero because they’re in class 2. Going off of this, $x_{1}$ and $x_{3}$ are correctly classified while $x_{2}$ is not correctly classified.

After this, I used the weight update rule from the linear perceptron algorithm, to find the value of the new weight w(1) based on the misclassified sample. I then plotted the new line corresponding to weight w(1), as well as the direction of the weight on the line.

Note: The update rule is $w(t + 1) = w(t) + y_{s}x_{s}$ , where $x_{s}$ and $y_{s} ∈ {−1, 1}$ is the feature and class label of misclassified sample s.

# w(1) = w(0) + ys * xs
# where:
# w(0) = [2, -1, 1]
# ys = -1
# xs = x2

# weight update rule to get w(1)
w1 = np.array(w0 - x2)
print("New weight w(1) =", w1)

New weight w(1) = [ 1 -4 -1]

# data for line corresponding to weight w(1)
x0 = np.linspace(-6,6)
y0 = 1-4*x0

# plot data points and weight line w(1) with direction
plot_graph(x, y, sample_names, colors, x0, y0, -1, -1, 'w(1)')

# inner product of weight line w(1) and data samples
inner_product(w1, samples, 'w1')

The inner product of w1 and x1: -6
The inner product of w1 and x2: -13
The inner product of w1 and x3: -16

Again, the inner product of w(1) and $x_{1}$ should be greater than zero because $x_{1}$ is part of class 1 and the inner products of w(1) and $x_{2}$ as well as w(1) and $x_{3}$ should both be less than zero because they’re in class 2. Going off of this, $x_{2}$ and $x_{3}$ are correctly classified while $x_{1}$ is not correctly classified.

Now, we use the rule $sign(w(t)^{T}x_{n})$ to run the linear perceptron algorithm until it converges, which in this case only takes one more iteration.

# weight update rule to get w(2)
w2 = np.array(w1 + x1)
print("New weight w(2) =", w2)

New weight w(2) = [ 2 -3  2]

# data for line corresponding to weight w(2)
x0 = np.linspace(0,5)
y0 = -1+(3/2)*x0

# plot data points and weight line w(2) with direction
plot_graph(x, y, sample_names, colors, x0, y0, -1, 1, 'w(2)')

# inner product of weight line w(2) and data samples
inner_product(w2, samples, 'w2')

The inner product of w2 and x1: 5
The inner product of w2 and x2: -3
The inner product of w2 and x3: -8

After the algorithm converges, all the samples are now correctly classified.

K-NN Classifier

2019-09-16T00:00:00+00:00

Classifying Benign vs Malignant Tumors Using K-NN Classifier

The main purpose of this project was to classify whether a tumor is benign or malign based on its attributes. This was done using data from the following UCI Machine Learning Repository: link The rows of these files represent the data samples, while columns 1-9 are the features and the class variable is column 10, as described below:

Clump Thickness: discrete values {1, 10}
Uniformity of Cell Size: discrete values {1, 10}
Uniformity of Cell Shape: discrete values {1, 10}
Marginal Adhesion: discrete values {1, 10}
Single Epithelial Cell Size: discrete values {1, 10}
Bare Nuclei: discrete values {1, 10}
Bland Chromatin: discrete values {1, 10}
Normal Nucleoli: discrete values {1, 10}
Mitoses: discrete values {1, 10}
Class: 2 for benign, 4 for malignant

# import data from .csv file into table
training_data = Table.read_table('hw1_question1_train.csv')
training_data

f1	f2	f3	f4	f5	f6	f7	f8	f9	label
10	10	9	3	7	5	3	5	1	4
1	1	1	1	1	1	3	1	1	2
1	1	1	1	1	1	3	1	1	2
5	1	1	1	1	1	3	1	1	2
8	10	10	10	5	10	8	10	6	4
8	10	8	8	4	8	7	7	1	4
1	1	1	1	2	1	3	1	1	2
10	10	10	10	7	10	7	10	4	4
10	10	10	10	3	10	10	6	1	4
8	7	8	7	5	5	5	10	2	4

... (473 rows omitted)

Data exploration: First, I used the training data to count how many samples were in the benign and malignant classes. After doing this it’s clear that the samples are not evenly distributed and that there’s roughly a 1/3 ratio of malignant to benign tumors. This means there is a slightly higher chance of a tumor being benign than it being malignant.

# count the total number of samples in the benign/malignant class 
num_benign = training_data.where('label', are.equal_to(2)).num_rows
num_malignant = training_data.where('label', are.equal_to(4)).num_rows

print('Number of samples belonging to benign class:', num_benign) 
print('Number of samples belonging to malignant class:', num_malignant) 

Number of samples belonging to benign class: 330
Number of samples belonging to malignant class: 153

Next, I plotted a histogram of each feature using the training data to see how they are distributed in the 1-10 range among the samples. (i.e., 9 total histograms). From this, you can see that the features are not distributed evenly in the 1-10 range. Feature 1 is the most evenly distributed, however the majority of the samples are still equal to or less than five. Overall, the samples seem to have a higher distribution on the lower numbers.

For the next step, I randomly select 5 pairs of features and plotted them on scatter plots. Each data point is color-coded to indicate the class in which the samples belong to (e.g., blue for benign, red for malignant). Looking at the results from this graph, the data appears to be seperable, with the benign samples being concentrated on the bottom left of the graphs and the malignant samples being centered on the top right of the graphs.

Implementing a K-Nearest Neighbor Classifier: Since the data does appear to seperable, K-NN is a valid model to use. If the data were not seperable into two groups, then K-NN would not be as good of a model for the data and would have a low classification accuracy. Now, to actually implement K-NN. In this project, I didn’t use any available libraries, instead I implemented the algorithm from scratch to better understand how it works. I also used two different distance metrics, the euclidean distance (l2-norm) and cosine similarity as distance measures to classify between the benign and malignant classes to see the difference in results.

###################### function definitions ######################
# calculates Euclidean distance (ED) btw two arrays of data

# the Euclidean distance is calculated by taking the square root
# of the sum of the squared differences of the arrays

# input: two arrays of numbers, arrays are same length

# output: returns the ED btw the two arrays of numbers 

def euclideanDistance(row1, row2):
    rows = np.subtract(row1,row2)
    distance = np.linalg.norm(rows)
    return distance 
    
# function to calculate cosine similarity for bonus

# input: two arrays of numbers, same length

# output: returns the cosine similarity between the two arrays

def cosineSimilarity(row1, row2):
    dot_prod = np.dot(row1, row2)
    norm1 = np.linalg.norm(row1)
    norm2 = np.linalg.norm(row2)
    denom = norm1 * norm2
    return dot_prod / denom


# function to get the K-Nearest Neighbors (k) in a set of arrays 
# (training_set) to one specific array (test_sample) by 
#     1) findining the Euclidean distance between each sample in 
#        the training_set and the test_sample and putting
#        them in a list called distances along with the
#        index of the corresponding row in the training_set
#     2) sort the list based on the distances (shortest to longest)
#     3) get the first k values from the distances list and return 
#        them in a list of tuples called neighbors 

# input: 1) an array of features of a sample who you want to find k 
#        neighbors for (test_sample)
#        2) a set of arrays of features for multiple samples who you 
#           will calculate the distance for against the testing 
#           array (training_set)
#        3) a value for k that will determine the number of closest 
#           samples that are returned

# output: a list of tuples of length k tha contains the row indexes 
#         in training arrays that are closest to the testing 
#         sample and the distance 

def getNeighbors(training_set, test_sample, k, dist_type):
    distances = []
    for i in range(training_set.num_rows):
        if dist_type == 'euclidean':
            d = euclideanDistance(training_set.row(i), test_sample)
        elif dist_type == 'cosine':
            d = cosineSimilarity(training_set.row(i), test_sample)
        elif dist_type == 'l0_norm':
            d = L0Norm(training_set.row(i), test_sample)
        distances.append((i, d))
    distances.sort(key=itemgetter(1))
    neighbors = []
    for i in range(k):
        neighbors.append(distances[i][0])
    return neighbors


# function to decide whether a sample belongs to class benign or 
# malignant based on its K-Nearest Neighbors by getting the label 
# of each neighbor, counting how many are benign and how many are 
# malignant and then returning whichever of the two classes had 
# the most votes

# input: list of tuples that has, index of row of neighbor and the 
# distance between the neighbor and sample we are trying to classify

# output: predicted label (either 2 or 4) of unlabeled sample

def getPredClass(neighbors):
    benign = 0
    malignant = 0
    for i in neighbors:
        if training_data.row(i)[9] == 4:
            malignant += 1
        else:
            benign += 1
    if benign > malignant:
        return 2
    else:
        return 4

For the next step, we want to figure out how well the K-NN model actually works on this dataset, so I wrote two functions to calculate both the accuracy and balanced accuracy of the model on the development set.

In addition to this, I explored the results with different values of k to see which resulted in better classification accuracies.

###################### function definitions ######################
# function to calculate the classification accuracy of the model, 
# accuracy is calculated as the number of correct classifications 
# over the total number of samples attempted to classify 

#input: array of actual label values and array of predicted label 
#       values

#output: accuracy of model

def getAcc(actual, pred):
    correct = 0
    for i in range(len(actual)):
        if actual[i] == pred[i]:
            correct += 1
    acc = (correct/(len(actual))) 
    return acc


# function to calculate the balanced classification accuracy of 
# the model, this is calculated as the number of correct 
# classifications of the first class over the total number of 
# samples in the first class summed by the number of classifications 
# of the second class ove the total number of samples in the second 
# class all divided by 2

#input: array of actual label values and array of predicted label 
#       values

#output: balanced accuracy of model

def getBalancedAcc(actual, pred):
    total_benign = np.count_nonzero(actual_label == 2)
    total_malignant = np.count_nonzero(actual_label == 4)
    correct_benign = 0
    correct_malignant = 0
    for i in range(len(actual)):
        if actual[i] == pred[i] and actual[i] == 2:
            correct_benign += 1
        elif actual[i] == pred[i] and actual[i] == 4:
            correct_malignant += 1
    bal_acc = ((correct_benign/total_benign) + (correct_malignant/total_malignant))/2
    return bal_acc

# setting up data for KNN abd Acc/BAcc calculations
no_labels_train = training_data.drop('label')            # remove labels from training data
labels_dev = Table().read_table('hw1_question2_dev.csv') # import dev data
no_labels_dev = labels_dev.drop('label')                 # remove labels from dev data
actual_label = labels_dev.column('label')                # put all dev data labels in array for Acc/BAcc check

# set up empty list to store values to put in table later
k_value = []
Acc_list = []
BAcc_list = []

# train model for values K = 1, 3, 5, 7, . . . , 19 using the train data
for k in range(1,20,2):
    predictions = []
    k_value.append(k)
    for i in range(no_labels_dev.num_rows):
        knn = getNeighbors(no_labels_train, no_labels_dev.row(i), k, 'euclidean')
        pred = getPredClass(knn)
        predictions.append(pred)
    # compute Acc and BAcc of model on dev set 
    Acc_list.append(getAcc(actual_label, predictions))
    BAcc_list.append(getBalancedAcc(actual_label, predictions))

    
# create table with all k-values and their corresponding Acc and BAcc values
model_test = Table().with_columns(
    'k-values', k_value,
    'Acc', Acc_list,
    'BAcc', BAcc_list)
model_test

k-values	Acc	BAcc
1	0.97	0.970856
3	0.99	0.991525
5	0.98	0.97933
7	0.97	0.967135
9	0.97	0.967135
11	0.97	0.967135
13	0.97	0.967135
15	0.97	0.967135
17	0.97	0.967135
19	0.97	0.967135

# plot the two metrics (Acc/BAcc) against the different values of K. 
model_test.plot('k-values')

According to the graph above, which plots the accuracy and balanced accuracy against the different values of k, the best hyper-parameter k based on the accuracy metric is 3 and the best hyper-parameter k based on the balanced accuracy is also 3. In this case, the best hyper-parameters are the same k value.

Next, I tested the model on the test set using the best hyper-parameters found previously and calculated the accuracy and balanced accuracy based on these k-values.

# setting up data for KNN
labels_test = Table().read_table('hw1_question2_test.csv') # import test data
no_labels_test = labels_test.drop('label')                 # remove labels from test data

# running KNN on data
predictions = []
for i in range(no_labels_test.num_rows):
    knn = getNeighbors(no_labels_train, no_labels_test.row(i), 3, 'euclidean')
    pred = getPredClass(knn)
    predictions.append(pred)
    
# getting accuracies 
actual_label = labels_test.column('label')
print("Model Accuracy: ", getAcc(actual_label, predictions))
print("Model Balanced Accuracy: ", getBalancedAcc(actual_label, predictions))

Model Accuracy:  0.95
Model Balanced Accuracy:  0.9324324324324325

The accuracies calculated on the test set are lower than the ones calculated on the development set, which is understandable and could be explained by the data in the development set being more similar to the data in the training set than the testing set data.

For the above steps, I’d been using the Euclidean distance, so I then used cosine similarity instead to see if the resulting accuracies were better or not.

# set up empty list to store values to put in table later
k_value = []
Acc_list = []
BAcc_list = []

# train model for values K = 1, 3, 5, 7, . . . , 19 using the train data
for k in range(1,20,2):
    predictions = []
    k_value.append(k)
    for i in range(no_labels_dev.num_rows):
        cos_sim = getNeighbors(no_labels_train, no_labels_dev.row(i), k, 'cosine')
        pred = getPredClass(cos_sim)
        predictions.append(pred)
    # compute Acc and BAcc of model on dev set 
    Acc_list.append(getAcc(actual_label, predictions))
    BAcc_list.append(getBalancedAcc(actual_label, predictions))

    
# create table with all k-values and their corresponding Acc and BAcc values
model_test = Table().with_columns(
    'k-values', k_value,
    'Acc', Acc_list,
    'BAcc', BAcc_list)
model_test

k-values	Acc	BAcc
1	0.61	0.489704
3	0.63	0.55577
5	0.61	0.551051
7	0.56	0.511369
9	0.6	0.537538
11	0.55	0.492278
13	0.55	0.503432
15	0.58	0.527242
17	0.58	0.527242
19	0.58	0.527242

# plot the two metrics (Acc/BAcc) against the different values of K. 
model_test.plot('k-values')

# running Cosine Similarity on data
predictions = []
for i in range(no_labels_test.num_rows):
    cos_sim = getNeighbors(no_labels_train, no_labels_test.row(i), 3, 'cosine')
    pred = getPredClass(cos_sim)
    predictions.append(pred)
    
# getting accuracies 
actual_label = labels_test.column('label')
print("Model Accuracy: ", getAcc(actual_label, predictions))
print("Model Balanced Accuracy: ", getBalancedAcc(actual_label, predictions))

Model Accuracy:  0.27
Model Balanced Accuracy:  0.2365937365937366

As you can see, using cosine similarity significanlty lowered the accuracies. The regular accuracy dropped from .93 to .27 and the balanced accuracy dropped from .93 to .24. This shows how important the distance metric used in K-NN is, as it can greatly determine whether the model is a good prediction model or not.

f1	f2	f3	f4	f5	f6	f7	f8	f9	label
10	10	9	3	7	5	3	5	1	4
1	1	1	1	1	1	3	1	1	2
1	1	1	1	1	1	3	1	1	2
5	1	1	1	1	1	3	1	1	2
8	10	10	10	5	10	8	10	6	4
8	10	8	8	4	8	7	7	1	4
1	1	1	1	2	1	3	1	1	2
10	10	10	10	7	10	7	10	4	4
10	10	10	10	3	10	10	6	1	4
8	7	8	7	5	5	5	10	2	4

f1	f2	f3	f4	f5	f6	f7	f8	f9	label
10	10	9	3	7	5	3	5	1	4
1	1	1	1	1	1	3	1	1	2
1	1	1	1	1	1	3	1	1	2
5	1	1	1	1	1	3	1	1	2
8	10	10	10	5	10	8	10	6	4
8	10	8	8	4	8	7	7	1	4
1	1	1	1	2	1	3	1	1	2
10	10	10	10	7	10	7	10	4	4
10	10	10	10	3	10	10	6	1	4
8	7	8	7	5	5	5	10	2	4

f1	f2	f3	f4	f5	f6	f7	f8	f9	label
10	10	9	3	7	5	3	5	1	4
1	1	1	1	1	1	3	1	1	2
1	1	1	1	1	1	3	1	1	2
5	1	1	1	1	1	3	1	1	2
8	10	10	10	5	10	8	10	6	4
8	10	8	8	4	8	7	7	1	4
1	1	1	1	2	1	3	1	1	2
10	10	10	10	7	10	7	10	4	4
10	10	10	10	3	10	10	6	1	4
8	7	8	7	5	5	5	10	2	4