HiveBrain v1.2.0
Get Started
← Back to all entries
patternpythonMinor

Reading and processing a file using Pandas

Submitted by: @import:stackexchange-codereview··
0
Viewed 0 times
readingfilepandasusingandprocessing

Problem

I am trying to read a file using pandas and then process it. For opening the file I use the following function:

import os
import pandas as pd

def read_base_file(data_folder, base_file):
    files = map(lambda x: os.path.join(data_folder, x), os.listdir(data_folder))
    if base_file in files:
        try:
            df = pd.read_csv(base_file, na_values=["", " ", "-"])
        except Exception, e:
            print "Error in reading", base_file
            print e
            df = pd.DataFrame()

    else:
        print "File Not Found."
        df = pd.DataFrame()
    return df


My main concerns are the if statement and what I should return if there is an error.

Solution

Generator expression

I advice using a generator expression instead of map:

map(lambda x: os.path.join(data_folder, x), os.listdir(data_folder))


should become:

(os.path.join(data_folder, x) for x  in os.listdir(data_folder))


Also x should be renamed to something more expressive.

Separation of concerns

You both print and return values, if the printing is for debugging purposes, use logger.log

Specific Exception

If you write:

except Exception, e:


any Exception will be caught, I suggest IOException.

Code Snippets

map(lambda x: os.path.join(data_folder, x), os.listdir(data_folder))
(os.path.join(data_folder, x) for x  in os.listdir(data_folder))
except Exception, e:

Context

StackExchange Code Review Q#97724, answer score: 4

Revisions (0)

No revisions yet.