patternpythonMinor
Split CSV by Repeated cells python
Viewed 0 times
repeatedcsvcellssplitpython
Problem
Any time that a row ID (oddly placed in column 8, i.e. row[7]) is repeated after the first instance, I want to write those rows into a second file. The code I'm using is extremely slow -- it's a 40-column CSV with about a million rows. This is what I have:
def in_out_gorbsplit(inf, outf1, outf2):
outf1 = csv.writer(open(outf1, 'wb'), delimiter=',', lineterminator='\n')
outf2 = csv.writer(open(outf2, 'wb'), delimiter=',', lineterminator='\n')
inf1 = csv.reader(open(inf, 'rbU'), delimiter=',')
inf1.next()
checklist = []
for row in inf1:
id_num = str(row[7])
if id_num not in checklist:
outf1.writerow(row)
checklist.append(id_num)
else:
outf2.writerow(row)Solution
Since
Also don't forget to close open file handles.
checklist is a list, a "not in" operation has to iterate over all elements to give the correct answer. In other words, it has a complexity of \$O(n)\$. Use a set() instead, to lower the complexity of the operation to \$O(1)\$, making it much faster. Also don't forget to close open file handles.
Context
StackExchange Code Review Q#71199, answer score: 3
Revisions (0)
No revisions yet.