patternpythonMinor
Converting latitude and longitude coordinates from CSV using web service
Viewed 0 times
coordinatescsvservicelongitudelatitudeusingwebandconvertingfrom
Problem
I am working on converting a mailing list that has longitude and latitude coordinates within the CSV file. This script I came up with does what I need, but this is my first real-world use of python. I want to know where I am making any mistakes, not using best-practices, and what can be optimized to make it faster.
The input.csv file has the following header:
And the script:
```
import sys
import os
import csv
import signal
import json
import urllib
import urllib2
import sqlite3
import codecs
import cStringIO
csv.field_size_limit(sys.maxsize)
class EmailList:
WEB_SERVICE_URL = 'http://open.mapquestapi.com/nominatim/v1/reverse.php?format=json'
def __init__(self, inputFile):
signal.signal(signal.SIGINT, self.signal_handler)
self.conn = None
self.initialize_database(inputFile)
self.convert_rows()
self.db_to_csv()
def signal_handler(self, signal, frame):
try:
self.conn.commit()
self.conn.close()
print '[DB changes committed and connection closed.]'
except sqlite3.ProgrammingError as e:
print '[script stopped]'
print e.message
sys.exit(0)
def initialize_database(self, file):
print 'checking for data.db...'
if not os.path.isfile('data.db'):
print 'data.db does not exist, converting csv to sqlite...'
with open(file) as inputFile:
reader = UnicodeReader(inputFile)
header = reader.next()
if self.conn is None:
self.conn = sqlite3.connect('data.db')
c = self.conn.cursor()
c.execute("DROP TABLE IF EXISTS email_list")
sql = """CREATE TABLE email_list (\n""" + \
",\n".join([("%s varchar" % name
The input.csv file has the following header:
"Email Address",MEMBER_RATING,OPTIN_TIME,OPTIN_IP,CONFIRM_TIME,CONFIRM_IP,LATITUDE,LONGITUDE,GMTOFF,DSTOFF,TIMEZONE,CC,REGION,LAST_CHANGED,LEID,EUIDAnd the script:
```
import sys
import os
import csv
import signal
import json
import urllib
import urllib2
import sqlite3
import codecs
import cStringIO
csv.field_size_limit(sys.maxsize)
class EmailList:
WEB_SERVICE_URL = 'http://open.mapquestapi.com/nominatim/v1/reverse.php?format=json'
def __init__(self, inputFile):
signal.signal(signal.SIGINT, self.signal_handler)
self.conn = None
self.initialize_database(inputFile)
self.convert_rows()
self.db_to_csv()
def signal_handler(self, signal, frame):
try:
self.conn.commit()
self.conn.close()
print '[DB changes committed and connection closed.]'
except sqlite3.ProgrammingError as e:
print '[script stopped]'
print e.message
sys.exit(0)
def initialize_database(self, file):
print 'checking for data.db...'
if not os.path.isfile('data.db'):
print 'data.db does not exist, converting csv to sqlite...'
with open(file) as inputFile:
reader = UnicodeReader(inputFile)
header = reader.next()
if self.conn is None:
self.conn = sqlite3.connect('data.db')
c = self.conn.cursor()
c.execute("DROP TABLE IF EXISTS email_list")
sql = """CREATE TABLE email_list (\n""" + \
",\n".join([("%s varchar" % name
Solution
Two things that pop out for me immediately:
See below:
Note, what that will do is attempt to use the slice operator on the dictionary. I just tried it now and it fails with a TypeError exception. What you want the code to look like is this:
Another thing I've noticed. Dealing with dictionaries, you're not quite sure how to handle getting "optional" parameters from them. Have a look at the refactored code I wrote for your json dict handling:
The get method on dictionary objects takes an optional second parameter that specifies a default value to return if the key is not found.
- You named a parameter "file", which is a built-in function in python.
- In one of your except blocks, you incorrectly use a dictionary.
See below:
converted['city': '']
converted['state': '']
converted['country': '']
converted['postal_code': '']Note, what that will do is attempt to use the slice operator on the dictionary. I just tried it now and it fails with a TypeError exception. What you want the code to look like is this:
converted['city'] = ''
converted['state'] = ''
converted['country'] = ''
converted['postal_code'] = ''Another thing I've noticed. Dealing with dictionaries, you're not quite sure how to handle getting "optional" parameters from them. Have a look at the refactored code I wrote for your json dict handling:
city = json_result['address'].get('city', '')
state = json_result['address'].get('state', '')
cc = json_result['address'].get('country_code', '')
postal_code = json_result['address'].get('postcode', '')The get method on dictionary objects takes an optional second parameter that specifies a default value to return if the key is not found.
Code Snippets
converted['city': '']
converted['state': '']
converted['country': '']
converted['postal_code': '']converted['city'] = ''
converted['state'] = ''
converted['country'] = ''
converted['postal_code'] = ''city = json_result['address'].get('city', '')
state = json_result['address'].get('state', '')
cc = json_result['address'].get('country_code', '')
postal_code = json_result['address'].get('postcode', '')Context
StackExchange Code Review Q#32400, answer score: 2
Revisions (0)
No revisions yet.