Memory Efficient Dataframes from MASSIVE SQL Queries!!!
Have you ever wondered how to create memory-efficient pandas dataframes with python? In this short I demonstrate just how easy it is by using panda's chunksize argument.
Here is the source code:
Libraries
import pandas as pd
import sqlalchemy
from sqlalchemy import create_engine
Creating SQL Alchemy engine object
engine = create_engine('postgresql://postgres:youtube@localhost:5432/eSports.val')
Initiailizing connection to database
conn = engine.connect()
SQL query as a string to pass into pd.read_sql function
sql = "SELECT * FROM players"
Demonstrating individual chunks
for chunk in pd.read_sql_query(sql, conn, chunksize=1000):
print(chunk)
Function to query in chunks and concat results into a final dataframe
def chunkDF(sql, conn, chunksize):
df = pd.read_sql_query(sql, conn, chunksize = chunksize)
final_df = pd.concat(df)
final_df.reset_index(drop=True, inplace=True)
return final_df
Passing the results of the function to a dataframe
df = chunkDF(sql, conn, 100)
On this page of the site you can watch the video online Memory Efficient Dataframes from MASSIVE SQL Queries!!! with a duration of hours minute second in good quality, which was uploaded by the user Data Science with Josh 27 November 2022, share the link with friends and acquaintances, this video has already been watched 3,963 times on youtube and it was liked by 122 viewers. Enjoy your viewing!
