Parsing and extracting data from XML documents using XPath and Python libraries for data analysis
How to handle XML?
XML = eXtensible Markup Language
Data are structured in a hierarchy of nodes
Nodes can be elements, text nodes or attributes
It can be used to represent structures with complex details
Key point: XML is self-descriptive - the tags describe the data they contain.
XML as a Tree Structure
Every XML document forms a tree hierarchy (like folders on your computer):
XML Document = Tree of Nodes
XML: An Example
<breakfast_menu>
<food>
<name>Belgian Waffles</name>
<price>$5.95</price>
<description>Two of our famous Belgian Waffles with...</description>
<calories>650</calories>
</food>
</breakfast_menu>
<breakfast_menu>: the root element
<food>: a child element
Belgian Waffles: a text content
XML: Targeting Elements with XPath
XPath is like a GPS for navigating XML documents:
XPath Navigation - Finding Data in XML
/ = one level down | // = search anywhere | [1] = first item only
XML: Read a Document Using Python (Native Option)
Python has built-in XML support:
import xml.etree.ElementTree as xmlReader
# read from xml
tree = xmlReader.parse('menu.xml')
root = tree.getroot()
# iterate through elements
for food in root.findall('food'):
name = food.find('name').text
print(name)
XML: Read a Document Using Python (with lxml)
from lxml import etree
# read from xml
tree = etree.parse('menu.xml')
root = tree.getroot()
print(root)
Why lxml? It has more extensive support of XPath and better performance than the native library.
XML: Get Elements Using XPath
elems = root.findall('./food')
data = [[elem.find("./name").text,
elem.find("./price").text
] for elem in elems]
print(data)
# [['Belgian Waffles', '$5.95'], ['French Toast', '$4.50']]
This creates a list of lists - perfect for DataFrame conversion!
XPath 101
When a query starts with /: looks from the root
Query starts with ./: path is relative to current position
Filter with predicates:
./food[starts-with(./name/text(), 'Be')]
Exercise: Load this XML file in your Python environment, then do the same in Orange using the Python Script widget.
Bind XML Results to an Orange Data Table
It is possible to bind "raw data" in Orange tables:
from Orange.data import *
data = [
['green', 4, 1.2, 'apple'],
['orange', 5, 1.1, 'orange'],
['yellow', 4, 1.0, 'peach']
]
color = DiscreteVariable('color', values=set([row[0] for row in data]))
calories = ContinuousVariable('calories')
fiber = ContinuousVariable('fiber')
fruit = DiscreteVariable('fruit', values=set([row[3] for row in data]))
domain = Domain([color, calories, fiber], class_vars=fruit)
table = Table.from_list(domain, data)
