Split latitude/longitude by degree to make file names and folder directory names

This code works and does exactly as I want, but I am wondering if/how it could be done faster/more efficiently? This runs quickly on a demo file, but bogs down when I introduce much larger files (3-100GB).

Premise:

Take a large list of latitudes and longitudes remove the negative values (WGS1984) and put them into a North, South, East, West space. Eliminate decimal places.

Use a shortened version of latitude and longitudes for directory creation.

Code:

```
import numpy as np
import pandas as pd
import h5py

hdf_file = '/path/to/file/lat_lon.h5'
ndir_root = '/output/path/where/directories/are/created/'
# split the file into two chunks to avoid memory error.
chunks = [0, 50, 100]
# open chunked file
for half in range(len(chunks) - 1):
hf = h5py.File(hdf_file, 'r')
print 'file loaded. reading meta lat/lon.'

meta_latitude = hf['latitude'][chunks[half]:chunks[half + 1]]
meta_longitude = hf['longitude'][chunks[half]:chunks[half + 1]]

meta_df = pd.DataFrame({'longitude': meta_longitude, 'latitude': meta_latitude})
print meta_df

# create an empty list to store coordinates
lat_list = []
lon_list = []
for lat in meta_df['latitude']:
lat_list.append(lat)
for lon in meta_df['longitude']:
lon_list.append(lon)

# take coordinates from list and convert to a numpy array
latitude_list = np.asarray(lat_list, dtype=float)
longitude_list = np.asarray(lon_list, dtype=float)

# convert numpy array to a string
lat_list_str = latitude_list.astype(str)

# make copies to avoid contamination in for loop processing
lat_list_str_copy_1 = lat_list_str
lat_list_str_copy_2 = lat_list_str

# remove negative values, add North, South, East West.
letter = set('-')

for lat_check in lat_list_str_copy_1:
if letter & set(lat_check):
lat_list_str3 = [i.replace('-', 'S') for i in lat_list_str_copy_1]
else:
lat_list_str3 = ['N' + lat_addchar for lat_addchar

Bug and accidental quadratic runtime

for lat_check in lat_list_str_copy_1:
    if letter & set(lat_check):
        lat_list_str3 = [i.replace('-', 'S') for i in lat_list_str_copy_1]
    else:
        lat_list_str3 = ['N' + lat_addchar for lat_addchar in lat_list_str_copy_1]

This code (which occurs with variations four times in your program) is probably the cause for the program becoming extremely slow for large input files:

Inside the for loop you want to format only the current latitude value, but instead the whole list is rewritten each time. If you have a list with n = 1000 items, the inner code (i.replace('-', 'S') or 'N' + lat_addchar) is executed n2 = 1 000 000 times, of which the first 999 000 times were wasted effort.

In addition, the result is wrong, because you either replace '-' by 'S' in all items, or prepend 'N' to all items, depending of the value of only the last item in the list.

Side note

letter = set('-')
if letter & set(lat_check):
    …

This is actually a clever idea, if you wanted to check if any of several letters is contained in a string. However, for a single letter, you can simply write the following:

if '-' in lat_check:
    …

(But checking for negative values can be done more easily, see below.)

Unused results

for i in lat:
    if len(i) < 4:
        i = i + '0'

for i in lon:
    if len(i) < 5:
        i = '0' + i

for i in short_lat:
    if i % 3 > 0:
        i -= 1
for i in short_lon:
    if i % 3 > 0:
        i -= 1

These have no effect and can simply be removed, because the resulting value of i is not used anywhere and is overwritten in the next loop iteration anyway.

meta_df['y'] = lat
meta_df['x'] = lon
meta_df['lon_list_str'] = lon_list_str
meta_df['lat_list_str'] = lat_list_str

This can be removed as well, because meta_df isn't used anywhere later on.

Since now the lists lat and lon aren't used anymore, all the computations leading to their creation can be removed, too. This involves the intermediate lists lat_list_str3 and lon_list_str3, as well as lat_list_str_copy_1 and lon_list_str_copy_1.

Assignments don't make copies

# make copies to avoid contamination in for loop processing
lat_list_str_copy_1 = lat_list_str
lat_list_str_copy_2 = lat_list_str
[…]
lon_list_str_copy_1 = lon_list_str
lon_list_str_copy_2 = lon_list_str

Besides lat_list_str_copy_1 and lon_list_str_copy_1 now not being used anymore, this doesn't do what you think:

These assignments only attach new names to the existing list objects, the do not create new copies of the lists!

However, making copies wouldn't have been necessary at all, because they were never modified.

You can just use the original name of the list wherever the _copy_1 or _copy_2 names are used.

This does create a new list, as you intended:

lat_list = []
for lat in meta_df['latitude']:
    lat_list.append(lat)

However, it could more simply be written as

lat_list = list(meta_df['latitude'])

(But it turns out that creating this list wasn't necessary either, see below.)

If you are astonished by how assignments don't make copies in Python, you can read a good explanation in this blog post.

Simplified approach

After these first steps of cleaning up, the way the coordinates take through your program becomes more apparent:

h5py.File →
pd.DataFrame →
list of numbers →
NumPy float array →
NumPy str array →
list of strings (actual processing: remove parts after decimal point) →
list of integers →
list of strings →
list of strings (actual processing: remove minus sign, add N/S/E/W prefix).

This chain is replicated for latitudes and for longitudes.

I hope you can agree with me that the number of intermediate steps seems excessive compared to the actual work done.

I suggest the following approach:

-
Treat (longitude, latitude) pairs as single objects.

-
Iterate directly over the contents of the h5py.File object, processing one coordinate at a time (using a generator), instead of storing many coordinates at once in lists or other containers. If h5py is implemented properly, this should lead to low memory consumption regardless of the input file size and make manual management of chunk processing unnecessary.

-
Perform only a single conversion, namely when formatting the numeric value of a coordinate as the string that is used as the name of a new directory.

-
Organize each task of which the program is composed into its own function. This has the following benefits:

• It is easier for future readers of the code to understand what the program does.

• It is easier to test whether the individual task is implemented correctly.

• It forces you to think more thoroughly about your program, leading to simpler code.

• New functionality can be added by adding more functions and reusing existing functions, instead of duplicating and modifying the same code, which is tedious and error-prone.

Here is

StackExchange Code Review Q#151690, answer score: 6