Python Tutorial: Handling missing values

Pubblicato il: 11 aprile 2020
sul canale di: DataCamp

1,910

Want to learn more? Take the full course at https://learn.datacamp.com/courses/de... at your own pace. More than a video, you'll learn hands-on coding & quickly apply skills to your daily work.

---
In the previous lesson you were introduced to the two null value types that you encounter in python. In this lesson, you will assign null values to the missing values in the dataset!

Missing values in a dataset aren't usually left unfilled, they are filled with dummy values like 'NA', '-' or '.' etc.

In this lesson, you will learn to detect such missing values as well as replace them with 'NaN'.

Let's use the 'college' dataset which contains various details of college students as an example. We'll load data using 'pd.read_csv()' of 'college.csv'.

The first step in analyzing the dataset is to read and print a snippet of the dataset. We'll print the head of the 'college' DataFrame.

Find that all columns have float values.

If you observe clearly, you can see that a few data points are filled with a period! This suggests that missing values might be represented by a period.

However, we can confirm this only through further analysis. We'll use the info() method to get a gist of the dataset.

Hey, somethings' odd here! All the columns except 'private' are of 'object' type although they are supposed to be float.

We can further explore and confirm by finding the unique values in one of the columns. This way we can find any non-numerical values!

Let's apply the '.unique()' method on the column 'csat' and sort them using 'np.sort()'. From the output you can clearly observe that '.' is the only string value present. Hence, we need to replace it with 'NaN'.

This can be simply achieved while loading the dataset to a DataFrame. You can use the argument 'na_values' in 'pd.read_csv' to specify the values for missing data.

If you again check the 'info()' of 'college', you'll find that all the columns are now 'float64' type. This is great!

Now, let's consider another dataset to detect hidden missing values.

We will use the Pima Indians Diabetes dataset which contains various clinical diagnostic information of the patients from the Pima community. While loading the dataset we can observe 'NaN' values for missing data when you print the head of the DataFrame.

As before, let's print the 'info()' of the 'diabetes' DataFrame. They are all 'float' or 'int' type as expected.

Further, we can analyze using the 'describe()' method on the 'diabetes' DataFrame.

Observe closely. Something very odd here is that the 'BMI' column has a minimum value of 0. But we are aware that BMI cannot be 0. Hence, the 0's must rather be missing values in disguise!

To confirm the same, we can filter all the rows where 'BMI' is 0. There are 11 rows which have BMI as 0. They must be missing values.

These types of missing values can be tricky as they require some level of domain knowledge.

We'll replace these 11 zeros of BMI column with 'NaN' and check again using 'np.isnan()' of diabetes.BMI.

Great! Now that we have successfully removed the hidden missing values and replaced them with 'NaN's, let's summarize what we learned in this lesson!

We learned to detect missing value characters like '.', detect the inherent missing values within the data like '0' and replace them with NaNs.

In the next lesson, you'll dig deeper

into analyzing the missing values. But it's now time to practice!

#PythonTutorial #DataCamp #Dealing #Missing #Data #Python

In questa pagina del sito puoi guardare il video online Python Tutorial: Handling missing values della durata di ore minuti seconda in buona qualità , che l'utente ha caricato DataCamp 11 aprile 2020, condividi il link con amici e conoscenti, su youtube questo video è già stato visto 1,910 volte e gli è piaciuto 13 spettatori. Buona visione!

109