Python Tutorial: Diagnose data for cleaning

Publicado el: 30 marzo 2020
en el canal de: DataCamp

459

Want to learn more? Take the full course at https://learn.datacamp.com/courses/cl... at your own pace. More than a video, you'll learn hands-on coding & quickly apply skills to your daily work.

---
Cleaning data is the process of preparing your dataset for analysis. It's important because data almost never comes in a way that does not need to be cleaned.

This chapter is going to cover how to diagnose your data for problems that require cleaning.

Here's some common data problems that you will learn how to clean.

Column names can have inconsistent capitalizations and/or bad characters.

Missing data needs to be identified and addressed.

Outliers can pose a potential problem and need to be investigated.

Duplicate rows can bias analysis and need to be found and dropped.

Untidy datasets can contain multiple problems, and prevent us from quickly transforming our dataset to one suitable for reporting, to a dataset that is suitable for analysis.

Other common data problems are processing columns before a column can be used for data analysis, and column types can signal the presence of unexpected data values.

We'll go into more detail on what these mean throughout the course.

This is a dataset that we've adapted from the European Environment Agency.

It gives the percent of females aged 15 or over who can read and write, the average number of children that a woman will give birth to, and population for a given country in a continent.

We have row indexes and column names.

An index in pandas is the basic object storing the row labels. You can reference indexes for rows and columns by their labels or by integer.

Looking at the first few rows can identify some of the common data problems already.

We have some column name inconsistencies.

Country and Continent are capitalized, female literacy has a space in it. Missing data, which is data that is not present for whatever reason. These are represented as the NaN values. And the country names are in French.

To load the data, we begin by importing the pandas library with the standard alias, pd.

The read_csv function takes a filename and loads it into a dataframe called df.

We can visually inspect the first and last 5 rows of our dataframe using head and tail methods, respectively.

Here we see the column name inconsistencies, missing data, and columns that need to be further processed.

If you have a lot of columns, calling the columns attribute of our dataframe will return an index of column names.

You see that the Country column actually has a space at the end of it. This was practically invisible when we looked at the head and tail of our data.

We can call the shape attribute to look at the number of rows and columns of our data. This is useful if you cannot visually see all your data with head and tail.

Our data has 164 rows. A quick Google search shows there are 196 countries in the world. So not every country in the world is listed in this dataset.

We can use the info method to get additional information about our dataframe.

This is a pandas dataframe object with 164 rows and 5 columns.

The population column has only 122 non-missing values, so we can subtract 122 from 164 and know there are 42 missing data points.

The fertility column should also contain numeric values, but it is stored as a object instead.

An object type is a generic type in pandas that is stored as a string.

Numeric datatypes will be represented as an int if there are no decimals, and a float if there are decimals.

We will want a closer look at these missing data points and wrong data types, but for now it's your turn to load a dataset and visually inspect it.

#PythonTutorial #DataCamp #Cleaning #Data #Python

En esta página del sitio puede ver el video en línea Python Tutorial: Diagnose data for cleaning de Duración hora minuto segunda en buena calidad , que subió el usuario DataCamp 30 marzo 2020, comparta el enlace con amigos y conocidos, en youtube este video ya ha sido visto 459 veces y le gustó 8 a los espectadores. Disfruta viendo!