Used Car Price Predictor

InterQuartile Range for 'price', 'odometer', and 'year' were visualized using the boxplots seen below. Any value considered more extreme than 1 ½ times the interquartile range above the third quartile or below the first quartile were deemed as outliers and eliminated. The box plots show that for prices, any listing amount whose log is 1 ½ times below 6.55 or above 11.55 are considered outliers. The box plot for odometer visually does not provide the interquartile range due to extreme outliers for this feature. These values were calculated and eliminated. The boxplot for year identifies 1996 as being the Q1 boundary for older vehicles. Since no vehicles newer than 2020/2021 exist, only extremely old vehicles could act as outliers for this variable.

We began with 458,213 rows and 25 columns of data and removed 62,231 rows and 7 columns to end up with 395,982 rows and 18 columns, 16 columns of which were utilized by ML algorithms.

Our data set contains 12 categorical variables and 4 numerical variables, excluding the price column. In order to apply the ML models, the categorical variables needed to be tranformed into numerical variables. The scikit-learn library LabelEncoder was applied for this purpose.

Since the data set is not normally distributed, all of the features have different ranges. Feature scaling is essential for ML algorithms that calculate distances between data. If not to scale, the feature with a higher value range starts dominating when calculating distances. Therefore, the range of all features should be normalized so that each feature contributes approximately proportionately to the final distance. The scikit-learn library MinMaxScalar was applied for this process.

During this step, 90% of the data was split as training data and remaining 10% used as test data.

In statistics, linear regression is a linear approach to modeling the relationship between a scalar response (dependent variable) and one or more explanatory variables (independent variables). In linear regression, the relationships are modeled using linear predictor functions whose unknown model parameters are estimated from the data. Such models are called linear models.

The performance of linear regression is determined by the differences between the actual values and predicted values. While the model presents a seemingly good fit, it only surmounted an R² score of 63%, thus other models may provide better prediction results. Additionally, linear regression considers 'year', 'odometer', 'fuel', and 'cylinders' to be the most important variables in predicting price as viewed in the feature importance graph.

Ridge Regression is a technique used for analyzing a multiple regression model that suffers from multicollinearity, or when more than two explanatory variables are highly linearly related. This commonly occurs in models with a large number of parameters. Ridge regression regularizes all parameters equally and provides improved efficiency in parameter estimation problems in exchange for a tolerable amount of bias. This is suited for multicollinearity, where ordinary least squares provide unbiased regression coefficients (maximum likelihood estimates as observed in the data set).

A yellowbrick library by Alpha Selection which uses cross validation found the best alpha value (20.336) to fit the data set. The way our variables interact to predict used car price was not suited to a Ridge model analysis which also gave a low R² score of 63%.

Lasso (least absolute shrinkage and selection operator) is a regression analysis method that performs both variable selection and regularization in order to enhance the prediction accuracy and interpretability of the resulting statistical model. It was originally formulated for linear regression models though Lasso regularization is easily extended to other statistical models including generalized linear models, generalized estimating equations, proportional hazards models, and M-estimators. Lasso’s ability to perform subset selection relies on the form of the constraint and has a variety of interpretations in terms of geometry, Bayesian statistics and convex analysis.

The Lasso model was not suited to our data set as indicated by a low R² score. After conducting our predictive analysis using three linear regression models, it appears that price is not impacted by any variable or variables in a clear linear fashion. We move on to other types of ML models for testing.

The k-nearest neighbors algorithm (k-NN) is a non-parametric classification method in which the input consists of the k closest training examples in the dat set. k-NN is a type of instance-based learning, or lazy learning, where the function is only approximated locally and all computation is deferred until function evaluation. In other words, it manipulates the training data and classifies the new test data based on distance metrics. The quality of the predictions depends on the distance measure, which if the features represent different physical units or come in vastly different scales then normalizing the training data can improve the models accuracy dramatically.

From both of the above figures, it can be observed that RMSLE value is at lowest when k is four. On the other hand, there is no signifcant difference between RMSLE values for when k is three through six. By choosing the lowest MSLE occurrence, the data set is trained with more consistency using neighbors_N is 5 and a 'euclidean distance' metric. Since we performed normalization on our data set beforehand, the algorithm was able to achieve an 80% R² score, which is a considerably higher accuracy rating for used car price predictions than what was produced by the linear regression models.

A Random Forest is a meta estimator that fits a number of classifying decision trees on various sub-samples of the data set and uses averaging to improve the predictive accuracy and control over-fitting. The sub-sample size is controlled with a max-samples parameter if by default (bootstrap=True), otherwise the whole dataset is used to build each tree. In random forests, each tree in the ensemble is built from a sample drawn with replacement (i.e., a bootstrap sample) from the training set. Furthermore, when splitting each node during the construction of a tree, the best split is found either from all input features or a random subset of size max features. Randomness helps to prevent over-fitting and decreases the variance of the forest estimator.

In our model, 180 trees were created with max features of 0.5. In general, the more the trees, the better the results. Random Forest produced an R² score of 90%. The variables importance diagram above shows that 'year' and 'odometer' are the variables with the highest degree of predictive usefulness out of all the variables. The performance diagram above shows that actual versus predicted MSLE's were nearly identical for all except 3 of the 25 instances.

A Bagging Regressor is an ensemble meta-estimator that fits base regressors each on random subsets of the original data set and then aggregates their individual predictions (either by voting or by averaging) to form a final prediction. Such a meta-estimator can typically be used as a way to reduce the variance of a black-box estimator (e.g., a decision tree), by introducing randomization into its construction procedure and then making an ensemble out of it.

In our model, DecisionTreeRegressor is used as the estimator with max depth of 20, which creates 50 decision trees that results in an R² score of 81%. Despite its complexity, the performance of Random Forest is much better than Bagging Regressor. The fundamental difference between these two ML models is that in Random Forests, only a subset of features are selected at random out of the total and the best split feature from the subset is used to split each node in a tree, unlike in bagging where all features are considered for splitting a node.

XGBoost is an ensemble learning method that is a specific implementation of the Gradient Boosted method which uses more accurate approximations to find the best tree model. Sometimes, it may not be sufficient to rely upon the results of just one machine learning model. Ensemble learning offers a systematic solution to combine the predictive power of multiple learners. The resultant is a single model which gives the aggregated output from several models. The models that form the ensemble, also known as base learners, could be either from the same learning algorithm or different

learning algorithms. Bagging (previously discussed) and boosting are two widely used ensemble learners. Though these two techniques can be used with several statistical models, the most predominant usage has been with decision trees. XGBoost employs scalability to drive fast learning through parallel and distributed computed, in addition to efficient memory usage. In order to fit the data set to the model, parameters were adjusted to provide cross validation of max depth 24, 200 decision trees (estimators) and a learning rate of 0.4.

Learning Rate: The most important hyperparameter when configuring a neural network, which controls how much to change the model in response to the estimated error each time the model weights are updated. Choosing the learning rate is challenging as a value too small may result in a long training process that could get stuck, whereas a value too large may result in learning a sub-optimal set of weights too fast or an unstable training process.

n_estimators: This is the number of trees you want to build before taking the maximum voting or averages of predictions. A higher number of trees give you better performance with the drawback being longer run times.

The algorithm produced the highest R² score at 91%. The performance chart above displays the great accuracy of the prediction model.