Handling Missing/Null Values

The dataset was checked for null/missing values. The dataset contains a total of 165,029 entries and 88 features and it was confirmed that the dataset does not contain missing or null values as its dimensions remained the same after processing.

Addition of Features

We added certain features which we would use in our statistical analysis, such as a binary column to indicate whether a household receives wages and another if they own or operate a business. We also added a column that computes the net total of a family every month, and each family's financial category.

Dropping Certain Features

However, we also dropped certain features that are redundant and not related to the study such as some of the elements in the breakdowns of the total income and a few nominal features.

Sampling the Data

Due to the huge volume of data, processing it would be time and resource heavy. So, we sampled 10% of the original volume which can be seen in the data section, leaving us with a sampled dataset containing 16,503 entries. We ensured that our sampled data is balanced as per the financial category feature, which means that in our sample there are 5501 entries of households per financial category.

In a nutshell...

This graph illustrates the average allocation of household income across key expenditure categories based on annual data from Filipino households incomes and expenditures. Notably, a significant portion of household income is allocated to food and water expenditures. This study aims to identify the expenditures that heavily affect a household's financial status and determine the effect of having businesses or employment wages to a households' overall financial status.

Feature Selection

To answer our primary question of identifying which factors most negatively affect Filipino households' finances the most, we utilized the Random Forest Classifier algorithm in a feature importance analysis. The graph below shows the importance value of the top ten identified features.

This data reveals that non-food expenditure, water supply, and bread, among others, seem to impact family finances the most. We can thus reject the null hypothesis and accept the alternative hypothesis as we have identified correlation between various factors and financial well-being among Filipino families.

Effect of Wages on Finances

In our analysis, we wanted to seek the answer to the question "Do households with regular wages coming from employment exhibit different patterns of negative income compared to those relying primarily other sources of income?"

To do so, we utilized a chi-squared test to see if there is a significant difference between households with and without wages. Here's what we got:

The p-value is less than 0.05. Hence, we reject the null hypothesis and accept the alternative hypothesis. This implies that there is a significant difference between households that have regular wages from employment and those who do not.

The number of households that have wages (employment income) in the high category is significantly greater than the number of households without wages. This implies that having employment income or wages in a household can help in improving their financial status. However, the financial category of a household can still be affected by multiple factors (mainly their expenditures) which explains why there are still households(with employment income) that are in the moderate and low financial category.

Effect of Business Ownership on Finances

We also wanted to answer the question "Do households with higher levels of entrepreneurial activities exhibit different patterns of negative income compared to those relying primarily on wage employment or other sources of income?"

To do so, we utilized a chi-squared test to see if there is a significant difference between households with and without a business or entrepreneurship. Here's what we got:

The p-value is less than 0.05. Hence, we reject the null hypothesis and accept the alternative hypothesis. This implies that there is a significant difference between households that have a business or entrepreneurial income and households that do not.

The number of households that have businesses or entrepreneurial activities in high category is greater than the number of households without businesses. This implies that having a business or doing entrepreneurial activities can help in improving their financial status. However, the financial category of a household can still be affected by multiple factors (mainly their expenditures and in this case their business losses) which explains why there are still households (with entrepreneurial activities) that are in the moderate and low financial category. Moreover, households' (that don't have entrepreneurial activities) financial category can also be improved by the presence of other sources of income which explains the number of households in the high category for those without entrepreneurial activities.

Testing out Different Predictor Models

The performance of Support Vector Machine (SVM), Recursive Feature Elimination (RFE), and Logistic Regression models in predicting the financial status of a household given their income and expenses were compared to identify the best model. To interpret the following figures, take note that we denoted high, moderate, and low financial statuses by the numbers 1, 0, and -1, respectively. Those that are of low financial status are most prone to the risk of experiencing the effects of poverty. Here's what we found:

Support Vector Machine Performance

A confusion matrix of the prediction performance of the SVM model is presented below, alongside its cross-validation and test accuracies.

Recursive Feature Elimination Performance

A confusion matrix of the prediction performance of the RFE model is presented below, alongside its cross-validation and test accuracies.

Logistic Regression Performance

A confusion matrix of the prediction performance of the Logistic Regression model is presented below, alongside its cross-validation and test accuracies.

Evidently, we can identify that logistic regression tends to yield the best results, as it scored the highest in both cross-validation and test accuracies. This performance could still be improved, however, using Grid Search to identify the best possible parameters for the highest possible accuracy of the model. The following section reveals the result of the improved model.

Improved Logistic Regression Model

A confusion matrix of the prediction performance of the Logistic Regression model with improved parameters as identified through Grid Search is presented below, alongside its cross-validation and test accuracies. This is clearly a substantial improvement over any of the earlier models presented.

Additionally, here's a t-SNE (t-distributed Stochastic Neighbor Embedding) visualization of the decision boundaries of our improved logistic regression model.

Using this model accurately predicts under what financial category you would most likely fall under: high (1), moderate (0), and low (-1). Those that are predicted to be classified under low financial category are at highest risk of experiencing poverty.