Recent Entries 10
- pattern minor 112d agoParsing Geographic Data From XML Files in PythonI have recently been working on a program which takes as input, log files generated from a running/biking app. These files are in an xml format which is incredibly regular. In my program, I first strip all of the geographic and time information, and output as a text file. Then, reading back this text file, instances of Entry class are compiled into a list. Using these entries, calculations (and eventually graphing and long term trends) can be achieved. This python script requires the library geopy to use, and has been tested on Python3. My main concern is how little of the code is pythonic in nature, with multiple if statements begging the question of optimization. I have uploaded a copy of an example input file for the script on my github, which can be found here. Typing '4-19-17a.bin' without the quotations at the prompt will begin parsing, and will print to standard output the string representation of each Entry instance that was created, along with a couple calculations. dataparse.py ``` import xml.etree.ElementTree as ET def processtimestring(time): year = int(time[:4]) month = int(time[5:7]) day = int(time[8:10]) h = int(time[11:13]) m = int(time[14:16]) s = float(time[17:-6]) return [year, month, day, h, m, s] def parsethedata(): time = [] lat = [] lng = [] alt = [] dist = [] garbage = '{http://www.garmin.com/xmlschemas/TrainingCenterDatabase/v2}' filestring = input('The file to parse: ') outputstring = filestring[:-4] + '.txt' f = open(outputstring, 'w') tree = ET.parse(filestring) root = tree.getroot() for child in root[0][0][1][4]: for info in child: if (not info.text): for position in info: f.write(position.tag.replace(garbage, '') + '\n') f.write(position.text + '\n') else: f.write(info.tag.replace(garbage, '') + '\n') f.write(info.text + '\
- snippet minor 112d agoConvert XML to CSVI'm pretty sure this code can be optimized, but I'm not talented enough in Linq to do it myself. Here's what I'm trying to do: I have an XML file that needs to be converted into a .csv file. The XML looks like this: ``` Super Mario Bros 14 29,99 -No Comment- N/A Nintendo Video Games 1985 001 The Legend of Zelda 12 34,99 -No Comment- N/A Nintendo Video Games 1986 002 ``` (There are many more Items in the list, but they are all the same.) The code I'm currently using is working as intended, here it is: ``` public void fileConvert_XMLToCSV() { //This method converts an xml file into a .csv file XDocument xDocument = XDocument.Load(FilePath_CSVToXML); StringBuilder dataToBeWritten = new StringBuilder(); var results = xDocument.Descendants("Item").Select(x => new { title = (string)x.Element("Name"), amount = (string)x.Element("Count"), price = (string)x.Element("Price"), year = (string)x.Element("Year"), productID = (string)x.Element("ProductID") }).ToList(); for (int i = 0; i < results.Count; i++) { string tempTitle = results[i].title; string tempAmount = results[i].amount; string tempPrice = results[i].price; string tempYear = results[i].year; string tempID = results[i].productID; dataToBeWritten.Append(tempYear); dataToBeWritten.Append(";"); dataToBeWritten.Append(tempTitle); dataToBeWritten.Append(";"); dataToBeWritten.Append(tempID); dataToBeWritten.Append(";"); dataToBeWritten.Append(tempAmount); dataToBeWritten.Append(";"); dataToBeWritten.Append(tempPrice); dataToBeWritten.Append(";"); dataToBeWritten.Append(0); dataToBeWritten.Append(";"); dataToBeWritten.Append(0); dataToBeWritt
- pattern minor 112d agoLoading configuration properties from XMLI would like a review of the transformation of the first method to the updated one below: ``` private static ArrayList getValuesfromXML(String key1, String key2) { ArrayList valueArray = new ArrayList(); try { String scriptPath = getScriptPath("prop.xml"); File file = new File(scriptPath); FileInputStream fileInput = new FileInputStream(file); Properties properties = new Properties(); properties.loadFromXML(fileInput); fileInput.close(); valueArray.add(properties.getProperty(key1)); valueArray.add(properties.getProperty(key2)); } catch (FileNotFoundException e) { e.printStackTrace(); } catch (IOException e) { e.printStackTrace(); } return valueArray; } ``` It looks as if it's GC'd so there's no memory leaks. And is not using a `finally` on the filehandle really an issue? ``` /** * read property values from the configuration file. Using standard Java Properties class. * * @param key1 * - Property key * @param key2 * - Property Key * @return - ArrayList values of properties based on keys */ private static ArrayList getPropertyValue (String key1, String key2){ final ArrayList valueArray = new ArrayList<>(); String scriptPath = getScriptPath(propertiesFileName); try (FileInputStream fileInput = new FileInputStream(scriptPath)) { Properties properties = new Properties(); properties.loadFromXML(fileInput); valueArray.add(properties.getProperty(key1)); valueArray.add(properties.getProperty(key2)); } catch (Exception ex) { ex.printStackTrace(); } return valueList; } ```
- pattern minor 112d agoParsing XML commands and parametersI'm C# programmer and started Python recently for a new project. This is a part of my project, and I want your opinion about my code and if it's clean or dirty, too long, has duplicate code, or it's good and standard. (cmd= command ... at the end of file is just for class testing) ``` import xml.etree.ElementTree as ET #TODO: processing attributes should be case insencitive class command: def __init__ (self,htmlcommand): self.preview=False self.members=[] self.__parse__(htmlcommand) def __parse__(self,htmlcommand): root = ET.fromstring(htmlcommand) self.service=root.attrib['service'] self.name=root.attrib['name'] self.source=root.attrib['source'] self.preview=bool(root.attrib['preview']) self.procedurename=root.attrib['procedurename'] mems=root.findall('member') if(len(mems)==0): raise Exception('command has no member') for m in mems: self.members.append(member(m)) class member: def __init__ (self,member): self.params=[] self.__parse__(member) def __parse__(self,member): self.name=member.attrib['name'] self.method=member.attrib['method'] try: self.order=member.attrib['order'] except: pass try: self.sort=member.attrib['sort'] except: pass self.__parseparams__(member) def __parseparams__(self,member): params=member.findall('./params/param') for p in params: self.params.append(param(p)) #def __getparams(member): class param: def __init__(self,param): self.name=param.attrib['name'] self.value=param.attrib['value'] cmd= command("""
- pattern minor 112d agoGenerating Settings Objects from Database via Factory ClassIn an ASP.NET MVC application (C#) have a factory-like class that generates settings objects of `ISettingType` for my application. However, some settings are very simple and a standard built-in such as a string or number type value can also suffice. I think that the way that I'm doing this is very much a code smell though. These settings are persisted within my database in a table that's acting as a key/value store. The key is a `NVARCHAR(50)` column being the name of the setting also acting as the primary key, and the value field is an `XML` column. Manipulating the XML directly is unnecessary, and I have various classes that implement the `ISettingType` interface, which includes methods for de/serialization from/to XML. As I mentioned some of these settings are very simple and only require a single string or number value. The factory looks like this: ``` public static class SettingsFactory { private const string MODEL_NAMESPACE = "Company.Models.Administration.SettingTypes."; public static ISettingType Create(ApplicationSetting appSetting) { object setting = DeserializeValue(appSetting); var settingType = setting as ISettingType; if (settingType != null) return settingType; throw new InvalidOperationException("Setting is not of a complex type requiring an ISettingType"); } public static T Create(ApplicationSetting appSetting) { if (typeof(ISettingType).IsAssignableFrom(typeof(T))) return (T)Create(appSetting); return (T) DeserializeValue(appSetting); } private static object DeserializeValue(ApplicationSetting appSetting) { XElement rootElement = XElement.Load(appSetting.SettingValue); if (rootElement.HasElements) { Type settingType = Type.GetType(MODEL_NAMESPACE + appSetting.SettingName); if (settingType != null) { using (XmlReader reader = rootElement.CreateReader())
- pattern minor 112d agoProgram to search many files and remove a certain stringThe reasons I create this program is because we had an issue of XML elements showing up with older assembly versions in, which were duplicate elements of the same element with the new assembly version in. Here is an example of the problematic files we have: ``` Version="dot_net_4_assembly_version_number", Version="dot_net_2_assembly_version_number", "identical_data_here" Version="dot_net_4_assembly_version_number", "identical_data_here" ``` So my program will be given a directory, and loop through all files in all sub-directories of that directory, and then if it matches with the regex to find the references to the older assemblies, it will remove that xml element. Here is the code: ``` class Program { static int totalChangedFilesCount; static int totalRemovedDotNetTwoReferences; static void Main(string[] args) { var allFilePaths = new List(); allFilePaths = GetAllFilePaths("C:/Temp/SolutionToTest"); foreach(var filePath in allFilePaths) { RemoveOldAssemblyReferencesFromFile(filePath); } Console.WriteLine($"Total files changed: {totalChangedFilesCount}"); Console.WriteLine($"Total .NET 2 refernces removed: {totalRemovedDotNetTwoReferences}"); Console.ReadLine(); } private static List GetAllFilePaths(string sourceDirectory, string filePattern = "*.*") { var filePaths = new List(); try { foreach (string file in Directory.EnumerateFiles(sourceDirectory, filePattern, SearchOption.AllDirectories)) { filePaths.Add(file); } } catch (Exception ex) { Console.WriteLine($"Error: {ex.Message}"); } return filePaths; } static void RemoveOldAssemblyReferencesFromFile(string filePath) { string xmlToClean = string.Empty; try { if (string.IsNullOrEmpty(filePath)) {
- pattern minor 112d agoParsing an XML file with elements for keys and valuesI'm parsing an XML file that is structured like this: ``` key value .... ``` My first code was using `.parse()` and `.getRoot()` but I had memory issues (1.5G memory use for 350M file). I solved my problem using an iterator and `.clear()` after reading this paper. ``` import xml.etree.ElementTree myDict = {} xmlIterator = xml.etree.ElementTree.iterparse('file.xml') while True: try: _ , elem1 = next(xmlIterator) _ , elem2 = next(xmlIterator) _ , _ = next(xmlIterator) except StopIteration: break myDict[elem1.text]=elem2 elem1.clear() elem2.clear() _.clear() ``` The issue is I need to access several children at the same time: - The first one is my key `` - The second one is the value I need `` - The third is not relevant `` I would like to know if this is a good practice, or if there is a better way to do that (better performance or readability). I also tried to follow the "Easier to ask for forgiveness than permission" principle with the exception block for ending the loop, but I'm not sure about multiple statements in the same exception block. In fact, I also don't understand why the third element I get with next() is ``.
- pattern minor 112d agoLoading items from XML documentI'm making a game in Unity and I need to have a inventory/item system, I decided to store my items in simple XML document which I later read from. I'm planning to have different varieties of items and they will all have distinct classes which extend they're base interface. Here's how my hierarchy looks like for now : IBaseItem ``` public interface IBaseItem { int ItemID { get; } string ItemName { get; } string Description { get; } } ``` A really simple interface which contains all the common properties between all the items. EquipementItem : IBaseItem ``` [System.Serializable] public abstract class EquipementItem : IBaseItem { public enum ItemTypes { None, Armor, Weapon } public enum SlotTypes { Head, Shoulders, Chest, Bracers, Gloves, Waist, Legs, Boots, Weapon } private readonly int itemID; public int ID { get { return itemID; } } private readonly string itemName; public string Name { get { return itemName; } } private readonly string description; public string Description { get { return description; } } private readonly ItemTypes itemType; public ItemTypes ItemType { get { return itemType; } } private readonly SlotTypes slotType; public SlotTypes SlotType { get { return slotType; } } protected EquipementItem(int itemID, string itemName, ItemTypes itemType, SlotTypes slotType, string description) { this.itemID = itemID; this.itemName = itemName; this.itemType = itemType; this.slotType = slotType; this.description = description; } } ``` This is the class for all the equippable items, and since the items are constant throughout the entire game I made the object immutable. However this might cause me some trouble since the constructor apparently needs too
- pattern minor 112d agoReading and working with a large number of XML attributesI'm currently facing the problem of unmarshalling a huge number (47) of attributes to a Java object. Changing the format of the XML file to be more structured is sadly not an option. The following should serve as an example. Imagine each request element has 47 attributes instead of two. ``` ``` So far I've found three possible solutions, but I'm not really happy with any of them. Context The XML file is refreshed minutely on the customers server and then polled and parsed by the client application. I have no influence on how it is generated or formatted. 1 Use a Bean This is probably the most naive solution, but this gets really boilerplate heavy with more than 600 lines for 47 attributes. On the other hand, this would probably be the easiest to parse. ``` import java.io.Serializable; import javax.xml.bind.annotation.XmlAttribute; import javax.xml.bind.annotation.XmlType; import javafx.beans.property.SimpleStringProperty; @XmlType(name = "Request") public class Request implements Serializable { private final SimpleStringProperty name = new SimpleStringProperty(); private final SimpleStringProperty id = new SimpleStringProperty(); public String getName() { return name.get(); } @XmlAttribute(name = "name") public void setName(String name) { this.name.set(name); } public SimpleStringProperty nameProperty() { return name; } public String getId() { return id.get(); } @XmlAttribute(name = "id") public void setId(String id) { this.id.set(id); } public SimpleStringProperty idProperty() { return id; } } ``` 2 Use a Map with String keys A bit more manageable boilerplate-wise, but still you'd require 47 constants for all the attribute names and it leaves you inflexible if you'd need something other than a `SimpleStringProperty`, a `SimpleDoubleProperty` for example. ``` package xml.model; import javafx.beans.property.SimpleStringP
- pattern minor 112d agoRecursively generating flat XML elements from hierarchical data structureMy class has a method that does some stuff, and then wants to recursively operate on some data. Specifically, I'm generating an XML report whose output includes a flat list of elements derived from a hierarchical data structure. Example Input (a Score) `{'wrapper':True, 'sub_scores':[ {'wrapper':True, 'sub_scores':[ {'wrapper':False, 'name':'count-objects', 'result':'pass', 'sub_scores':[]}, {'wrapper':False, 'name':'find-object', 'result':'pass', 'sub_scores':[ {'wrapper':False, 'name':'measure.x', 'result':'pass', 'sub_scores':[]}, {'wrapper':False, 'name':'measure.y', 'result':'pass', 'sub_scores':[]}, ]}, {'wrapper':False, 'name':'find-object', 'result':'fail', 'sub_scores':[ {'wrapper':False, 'name':'measure.x', 'result':'skip', 'sub_scores':[]}, {'wrapper':False, 'name':'measure.y', 'result':'skip', 'sub_scores':[]}, ]} ]} ]} ` Desired Output ` ` In JavaScript or Lua I would create a helper function within my method that does the recursion. In Ruby I would create a lambda or proc that does the recursion. Is the 'right' way to do this in Python to create a helper static function? Or is there a better way to keep the original method self-contained, perhaps allowing closures over local variables? ``` class Scorer: def generate_report(self, score): r = etree.Element('report', {'user': getpass.getuser(), 'time': timestamp()}) etree.SubElement(r, 'summary', {'tests': score.total(), 'pass': score.passed(), 'skip': score.passed(), 'fail': score.failed()}) etree.SubElement(r, 'results', {'file': self.results_path}) samples = etree.SubElement(r, 'samples') Scorer._add_score_to_samples(samples, score) return r @staticmethod def _add_score_to_samples(samples, score): # Some scores are wrappers that should not be included in output if not score.wrapper: