How to read a CSV file in Python

Reading CSV files in Python enables you to work with structured data stored in comma-separated values format. The Python standard library includes powerful tools like pandas and the built-in csv module to efficiently process these files.

This guide covers essential techniques for handling CSV data in Python. All code examples were created with Claude, an AI assistant built by Anthropic, to demonstrate practical implementations and common debugging solutions.

Reading CSV files with the csv module

import csv
with open('data.csv', 'r') as file:
    csv_reader = csv.reader(file)
    for row in csv_reader:
        print(row)

['Name', 'Age', 'City']
['John', '28', 'New York']
['Mary', '24', 'Boston']

The csv.reader() function creates an iterator that processes each row of your CSV file as a list of strings. This approach provides granular control over data processing while maintaining memory efficiency with large files.

Python's built-in csv module handles common CSV parsing challenges automatically. You'll benefit from:

• Proper handling of quoted fields containing commas
• Automatic line ending detection across operating systems
• Memory-efficient row-by-row processing

The with statement ensures proper file handling by automatically closing the file after processing. This prevents resource leaks and data corruption that could occur if the program exits unexpectedly.

Basic CSV handling techniques

Beyond the basic csv module, Python offers additional tools and techniques to handle CSV files with greater flexibility and intuitive data access.

Using pandas to read CSV files

import pandas as pd
df = pd.read_csv('data.csv')
print(df.head())

Name  Age      City
0  John   28  New York
1  Mary   24    Boston

The pandas library simplifies CSV handling by creating a DataFrame—a powerful table-like data structure. With just one line of code, pd.read_csv() loads your entire CSV file into memory and automatically detects column names and data types.

• The DataFrame provides intuitive data access through column names instead of numeric indices
• The head() function displays the first few rows of data, helping you quickly verify the import
• Column operations and data filtering become significantly easier compared to the basic csv module

While pandas consumes more memory than row-by-row processing, it excels at data analysis tasks and handles complex CSV files with features like missing value detection and custom delimiter support.

Reading CSV with different delimiters

import csv
with open('data.csv', 'r') as file:
    csv_reader = csv.reader(file, delimiter=';')
    for row in csv_reader:
        print(row)

['Name', 'Age', 'City']
['John', '28', 'New York']
['Mary', '24', 'Boston']

Not all CSV files use commas as separators. The delimiter parameter in csv.reader() lets you specify a different character to split your data. In this example, semicolons separate the values instead of commas.

• Common delimiters include semicolons (;), tabs (\t), and pipes (|)
• European datasets often use semicolons because commas serve as decimal separators in those regions
• The code processes the file exactly as before. The only change is telling Python which character marks the boundary between fields

You can verify the correct delimiter by opening your CSV file in a text editor. The wrong delimiter will result in improperly split data or the entire row appearing as a single field.

Using csv.DictReader for column access

import csv
with open('data.csv', 'r') as file:
    csv_reader = csv.DictReader(file)
    for row in csv_reader:
        print(f"Name: {row['Name']}, City: {row['City']}")

Name: John, City: New York
Name: Mary, City: Boston

The DictReader class transforms each CSV row into a dictionary, making your data more accessible through column names instead of numeric indices. This approach eliminates the need to track column positions manually, reducing errors in your code.

• Access values using column names as dictionary keys: row['Name'] instead of row[0]
• The first row of your CSV automatically becomes the dictionary keys unless you specify custom ones
• Column names remain consistent even if you reorder the CSV columns

This method particularly shines when working with CSVs that have many columns or when you need to access only specific fields. The code becomes more readable and maintainable since column references clearly indicate which data you're processing.

Advanced CSV processing

Building on these foundational CSV techniques, Python offers powerful methods to selectively process columns, handle large datasets efficiently, and manage data quality issues in your files.

Reading specific columns from CSV

import csv
with open('data.csv', 'r') as file:
    csv_reader = csv.reader(file)
    header = next(csv_reader)
    name_index = header.index('Name')
    for row in csv_reader:
        print(f"Name: {row[name_index]}")

Name: John
Name: Mary

This code demonstrates how to extract specific columns from a CSV file without loading unnecessary data. The next() function reads the first row as the header, enabling you to find column positions dynamically using header.index().

• The name_index variable stores the position of the 'Name' column. This approach makes your code more resilient to changes in column order
• Using row[name_index] retrieves only the name field from each row instead of processing all columns
• This method proves especially valuable when working with large CSV files containing many columns you don't need

The f-string formatting creates clean, readable output by displaying just the name values. This selective reading technique optimizes memory usage and processing speed for your specific data needs.

Reading large CSV files efficiently

def read_in_chunks(file_path, chunk_size=1000):
    with open(file_path, 'r') as file:
        reader = csv.reader(file)
        header = next(reader)
        chunk = []
        for i, row in enumerate(reader):
            if i % chunk_size == 0 and i > 0:
                yield chunk
                chunk = []
            chunk.append(row)
        yield chunk

for chunk in read_in_chunks('large_data.csv'):
    print(f"Processing {len(chunk)} rows...")

Processing 1000 rows...
Processing 1000 rows...
Processing 578 rows...

The read_in_chunks() function processes large CSV files by breaking them into smaller, manageable pieces called chunks. This approach prevents memory overload when handling massive datasets that won't fit into RAM all at once.

• The function uses Python's yield keyword to create a generator that returns one chunk at a time
• Each chunk contains chunk_size rows (defaulting to 1000) from the CSV file
• The enumerate() function tracks row position while % operator determines when to yield the current chunk

This chunked reading pattern enables efficient processing of CSV files that could be gigabytes in size. The code processes each chunk independently before moving to the next one. This keeps memory usage constant regardless of file size.

Handling missing values in CSV files

import pandas as pd
import numpy as np

df = pd.read_csv('data_with_missing.csv')
df.fillna({'Name': 'Unknown', 'Age': 0, 'City': 'Not specified'}, inplace=True)
print(df.head())

Name  Age          City
0    John   28      New York
1    Mary   24        Boston
2  Unknown   35  Not specified

Missing values in CSV files can corrupt your data analysis. The pandas library provides robust tools to handle these gaps efficiently. The fillna() method replaces empty values with specified defaults for each column.

• The dictionary passed to fillna() maps column names to their default values
• Setting inplace=True modifies the DataFrame directly instead of creating a copy
• Common default values include zero for numeric fields and descriptive text for strings

This approach maintains data consistency while clearly marking which values were originally missing. You can easily track these substitutions later by searching for the default values you specified.

Calculating sales statistics from csv data

The csv module enables rapid calculation of key business metrics like total revenue and average transaction value from your sales data files.

import csv

total_sales = 0
count = 0
with open('sales.csv', 'r') as file:
    reader = csv.DictReader(file)
    for row in reader:
        total_sales += float(row['Amount'])
        count += 1

print(f"Total sales: ${total_sales:.2f}")
print(f"Average sale: ${total_sales/count:.2f}")

This code calculates the total and average sales from a CSV file containing transaction data. The DictReader processes each row as a dictionary, making it easy to access the 'Amount' column by name. The script maintains two running counters: total_sales accumulates the sum while count tracks the number of transactions.

• The float() function converts string amounts to numbers for mathematical operations
• F-strings format the output with two decimal places using :.2f
• The average calculation happens only once at the end, dividing total by count

This approach efficiently processes large sales datasets with minimal memory usage since it reads one row at a time.

Merging data from multiple csv sources

Python's csv module enables you to combine data from separate CSV files into enriched datasets by matching records across common identifiers like customer IDs or transaction numbers.

import csv

# Load customer data dictionary
customers = {}
with open('customers.csv', 'r') as file:
    for row in csv.DictReader(file):
        customers[row['id']] = row['name']

# Create enriched order report
with open('orders.csv', 'r') as in_file, open('report.csv', 'w', newline='') as out_file:
    reader = csv.DictReader(in_file)
    writer = csv.writer(out_file)
    
    writer.writerow(['order_id', 'customer', 'amount'])
    for order in reader:
        customer = customers.get(order['customer_id'], 'Unknown')
        writer.writerow([order['id'], customer, order['amount']])

print("Generated report with customer information")

This code creates a customer-enriched order report by combining data from two CSV files. First, it builds a dictionary that maps customer IDs to names from customers.csv. The DictReader makes accessing columns by name straightforward.

The second part reads orders.csv and writes a new report with enhanced customer information. The customers.get() method safely retrieves customer names using order IDs—returning "Unknown" if an ID isn't found. The script processes orders one at a time to maintain memory efficiency.

• Uses dictionary lookup for fast customer name retrieval
• Handles missing customer data gracefully
• Creates a clean report with just the essential fields: order ID, customer name, and amount

Common errors and challenges

Python developers frequently encounter encoding issues, data type mismatches, and CSV formatting challenges that can disrupt their data processing workflows.

Fixing UnicodeDecodeError when reading CSV files with special characters

CSV files containing non-English characters often trigger a UnicodeDecodeError when Python attempts to read them with default encoding settings. This common issue affects developers working with international datasets or text containing special characters.

import csv
with open('international_data.csv', 'r') as file:
    csv_reader = csv.reader(file)
    for row in csv_reader:
        print(row)

The code fails because it assumes ASCII or UTF-8 encoding. When the file contains special characters encoded differently, Python can't properly decode them. The following code demonstrates the proper way to handle this scenario.

import csv
with open('international_data.csv', 'r', encoding='utf-8') as file:
    csv_reader = csv.reader(file)
    for row in csv_reader:
        print(row)

Adding the encoding='utf-8' parameter when opening files ensures Python correctly interprets special characters and international text. This solution prevents the UnicodeDecodeError that commonly occurs with non-English content.

• Watch for this error when processing data containing accented letters, Chinese characters, or emojis
• Common file sources include exported spreadsheets from European or Asian systems
• If UTF-8 doesn't work, try other encodings like 'latin-1' or 'cp1252' based on your data's origin

You can identify potential encoding issues by examining your data source's geographic origin or checking if it contains special characters before processing.

Converting string values to numbers in CSV data

CSV files store all data as text strings. When you try to perform mathematical operations on numeric columns, Python raises a TypeError. The code below demonstrates this common pitfall when attempting to add price values directly from CSV rows without proper type conversion.

import csv
with open('prices.csv', 'r') as file:
    csv_reader = csv.reader(file)
    next(csv_reader)  # Skip header
    total = 0
    for row in csv_reader:
        total += row[1]  # Trying to add price directly
print(f"Total: {total}")

The code fails because row[1] returns a string value. Adding strings with the += operator concatenates them instead of performing numerical addition. The following code demonstrates the proper way to handle numeric CSV data.

import csv
with open('prices.csv', 'r') as file:
    csv_reader = csv.reader(file)
    next(csv_reader)  # Skip header
    total = 0
    for row in csv_reader:
        total += float(row[1])  # Convert string to float before adding
print(f"Total: {total}")

The solution converts string values to floating-point numbers using the float() function before performing arithmetic operations. This prevents Python from treating numeric data as text strings and attempting string concatenation instead of mathematical addition.

• Watch for this issue when processing CSV columns containing prices, quantities, or measurements
• Consider using int() for whole numbers or decimal.Decimal() for precise financial calculations
• Add error handling to manage invalid numeric strings that might appear in your data

The float() conversion works well for most numeric data. However, be cautious with currency values where floating-point precision could lead to rounding errors in calculations.

Handling quoted text in CSV files with the quoting parameter

CSV files containing text with embedded commas require special handling to parse correctly. The csv.reader() function can misinterpret quoted fields as separate values, splitting them incorrectly. The following code demonstrates this common parsing challenge.

import csv
with open('data_with_quotes.csv', 'r') as file:
    csv_reader = csv.reader(file)
    for row in csv_reader:
        print(row)

The code fails to handle fields containing commas within quoted text, causing incorrect data splitting. For example, a field like "Smith, John" splits into two separate values instead of staying as one name. Let's examine the corrected approach in the code below.

import csv
with open('data_with_quotes.csv', 'r') as file:
    csv_reader = csv.reader(file, quoting=csv.QUOTE_NONNUMERIC)
    for row in csv_reader:
        print(row)

The quoting=csv.QUOTE_NONNUMERIC parameter tells Python to respect quoted text fields as single values, preventing incorrect splitting at commas within quotes. This solves parsing issues with fields like "Smith, John" that should remain unified.

• Watch for this issue when your CSV contains addresses, names, or descriptions with embedded commas
• The parameter also automatically converts unquoted numeric values to floats
• Common data sources include exported contact lists or product catalogs with detailed descriptions

Without proper quote handling, your data processing could silently create errors by splitting fields incorrectly. Always verify your CSV structure and field contents before processing.