Python and XML (and Google!) in publishing applications

IBM Developer Works has just released an article of mine on High-Performance XML Parsing in Python. Although there is nothing publishing-centric about the article itself, it was based on my own experience in dealing with large XML datasets in academic publishing.

Massive XML files are uncommon in the general web development world, where the primary roles of XML are either as configuration files, read only infrequently, or for interchange across the web, in which case the files are necessarily small. It’s rare to encounter XML measured in gigabytes or more; data at that level is usually stored in a relational database.

For that reason I find myself frustrated with many XML tools, even those ostensibly designed to handle large amounts of data. Too often they don’t scale well or at least easily. I don’t believe that scaling should be a black art that each individual developer needs to solve independently. Unfortunately, in commercial products ease-of-use is a key bullet point and computationally-difficult problems are hard to summarize in a user’s guide.

Read More..

Open Source XML Database

As quoted from Wikipedia, an XML database is a data persistence software system that allows data to be stored in XML format. This data can then be queried, exported and serialized into any format the developer wishes.

Two major classes of XML database exist:

1. XML-enabled: these map all XML to a traditional database (such as a relational database), accepting XML as input and rendering XML as output. This term implies that the database does the conversion itself (as opposed to relying on middleware).

2. Native XML (NXD): the internal model of such databases depends on XML and uses XML documents as the fundamental unit of storage, which are, however, not necessarily stored in the form of text files.

Read more..

Open Source XML Database

As quoted from Wikipedia, an XML database is a data persistence software system that allows data to be stored in XML format. This data can then be queried, exported and serialized into any format the developer wishes.

Two major classes of XML database exist:

XML-enabled: these map all XML to a traditional database (such as a relational database), accepting XML as input and rendering XML as output. This term implies that the database does the conversion itself (as opposed to relying on middleware).
Native XML (NXD): the internal model of such databases depends on XML and uses XML documents as the fundamental unit of storage, which are, however, not necessarily stored in the form of text files. Read more..