patternpythonMinor
Merging multiple JSON files using Python
Viewed 0 times
mergingfilesusingmultiplepythonjson
Problem
I have multiple (1000+) JSON files each of which contain a JSON array. I want to merge all these files into a single file.
I came up with the following, which reads each of those files and creates a new object with all the contents. I then write this new object into a new file.
Is this approach efficient? Is there a better way to do so?
I came up with the following, which reads each of those files and creates a new object with all the contents. I then write this new object into a new file.
Is this approach efficient? Is there a better way to do so?
head = []
with open("result.json", "w") as outfile:
for f in file_list:
with open(f, 'rb') as infile:
file_data = json.load(infile)
head += file_data
json.dump(head, outfile)Solution
- First off, if you want reusability, turn this into a function. The function should have it's respective arguments.
- Secondly, instead of allocating a variable to store all of the JSON data to write, I'd recommend directly writing the contents of each of the files directly to the merged file. This will help prevent issues with memory.
- Finally, I just have a few nitpicky tips on your variable naming. Preferably,
headshould have a name more along the lines ofmerged_files, and you shouldn't be usingfas an iterator variable. Something likejson_filewould be better.
Context
StackExchange Code Review Q#87254, answer score: 7
Revisions (0)
No revisions yet.