Case-study of Machine Learning Hyperparameter Tuning to check Exponential change in Classification and Regression models accuracy

Introduction

Hyperparameter tuning plays a crucial role in optimizing the performance of machine learning models. In this case study, we explore the impact of hyperparameter tuning on model accuracy using various supervised learning algorithms for classification and regression tasks. The dataset used in this study is the Holiday Package Prediction Dataset, where the goal is to predict customer preferences for holiday packages based on various features.

This study aims to:

• Compare different machine learning models before and after hyperparameter tuning.
• Analyze the exponential improvement in model accuracy.
• Provide comparative code snippets and results for each model.

Datasets Overview

Dataset for Classification

The Holiday Package Prediction Dataset consists of multiple features including customer demographics, travel history, budget preferences, and past package purchases. The target variable is whether a customer will book a holiday package (classification) and the predicted expenditure on holiday packages (regression).

Holiday Package Prediciton

1) Problem statement. “Trips & Travel.Com” company wants to enable and establish a viable business model to expand the customer base. One of the ways to expand the customer base is to introduce a new offering of packages. Currently, there are 5 types of packages the company is offering * Basic, Standard, Deluxe, Super Deluxe, King. Looking at the data of the last year, we observed that 18% of the customers purchased the packages. However, the marketing cost was quite high because customers were contacted at random without looking at the available information. The company is now planning to launch a new product i.e. Wellness Tourism Package. Wellness Tourism is defined as Travel that allows the traveler to maintain, enhance or kick-start a healthy lifestyle, and support or increase one’s sense of well-being. However, this time company wants to harness the available data of existing and potential customers to make the marketing expenditure more efficient.

2) Data Collection. The Dataset is collected from https://www.kaggle.com/datasets/susant4learning/holiday-package-purchase-prediction The data consists of 20 column and 4888 rows.

Dataset for regression

Used Car Price Prediction

1) Problem statement. * This dataset comprises used cars sold on cardehko.com in India as well as important features of these cars. * If user can predict the price of the car based on input features. * Prediction results can be used to give new seller the price suggestion based on market condition.

2) Data Collection. * The Dataset is collected from scrapping from cardheko webiste * The data consists of 13 column and 15411 rows.

Supervised Learning Models Used

We will apply hyperparameter tuning to the following classification and regression models:

Classification Models

1. Logistic Regression
2. Decision Tree Classifier
3. Random Forest Classifier
4. Gradient Boosting Classifier
5. AdaBoost Classifier
6. XGBoost Classifier
7. Support Vector Machine (SVM)
8. k-Nearest Neighbors (KNN)

Regression Models

1. Linear Regression
2. Decision Tree Regressor
3. Random Forest Regressor
4. Gradient Boosting Regressor
5. AdaBoost Regressor
6. XGBoost Regressor
7. Support Vector Regressor (SVR)
8. k-Nearest Neighbors (KNN) Regressor

Methodology: Hyperparameter Tuning

For each model, we use RandomizedSearchCV to find the best hyperparameters and analyze their impact on model accuracy.

Before Hyperparameter Tuning

from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score

# Load dataset
df = pd.read_csv("holiday_package.csv")
X = df.drop("target", axis=1)
y = df["target"]

# Train-test split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Train model
model = RandomForestClassifier()
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

# Evaluate model
print("Accuracy:", accuracy_score(y_test, y_pred))

After Hyperparameter Tuning

from sklearn.model_selection import RandomizedSearchCV

# Define parameter grid
param_grid = {
    'n_estimators': [50, 100, 200],
    'max_depth': [None, 10, 20, 30],
    'min_samples_split': [2, 5, 10],
    'min_samples_leaf': [1, 2, 4]
}

# Perform Randomized Search
random_search = RandomizedSearchCV(RandomForestClassifier(), param_distributions=param_grid, n_iter=50, cv=5, verbose=2, n_jobs=-1)
random_search.fit(X_train, y_train)

# Evaluate best model
y_pred_tuned = random_search.best_estimator_.predict(X_test)
print("Tuned Accuracy:", accuracy_score(y_test, y_pred_tuned))

Comparative Results: Pre-Tuning vs. Post-Tuning

Observations

• Hyperparameter tuning significantly improves model accuracy.
• Boosting algorithms such as XGBoost and Gradient Boosting show exponential improvement.
• Random Forest benefits highly from parameter tuning, showing increased generalization.
• SVM and KNN, while improving, do not show exponential changes compared to tree-based models.

Conclusion

This case study demonstrates how hyperparameter tuning can lead to exponential improvement in model accuracy. Using RandomizedSearchCV, we identified optimal parameters, leading to significant accuracy gains. The findings suggest that investing in hyperparameter tuning is crucial for achieving the best predictive performance in machine learning models.

Future Work

• Apply Bayesian Optimization for tuning.
• Explore deep learning models for holiday package prediction.
• Test hyperparameter tuning using GPU acceleration for faster training.

This study reinforces the importance of hyperparameter tuning and provides a practical approach to achieving optimal model performance.