Take the 2-minute tour ×
Ask Ubuntu is a question and answer site for Ubuntu users and developers. It's 100% free, no registration required.

I'm trying to split up a 13GB xml file into small ~50MB xml files with this XSLT style sheet.

But this process kills xsltproc after I see it taking up over 1.7GB of memory (that's the total on the system).

Is there any way to deal with huge XML files with xsltproc? Can I change my style sheet? Or should I use a different processor? Or am I just S.O.L.?

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0"
                xmlns:exsl="http://exslt.org/common"
                extension-element-prefixes="exsl"
                xmlns:fn="http://www.w3.org/2005/xpath-functions">
<xsl:output method="xml" indent="yes"/>
<xsl:strip-space elements="*"/>
<xsl:param name="block-size" select="75000"/>

<xsl:template match="/">
        <xsl:copy>
                <xsl:apply-templates select="mysqldump/database/table_data/row[position() mod $block-size = 1]" />
        </xsl:copy>
</xsl:template>

<xsl:template match="row">
        <exsl:document href="chunk-{position()}.xml">
        <add>
                <xsl:for-each select=". | following-sibling::row[position() &lt; $block-size]" >
                <doc>
                        <xsl:for-each select="field">
                        <field>
                                <xsl:attribute name="name"><xsl:value-of select="./@name"/></xsl:attribute>
                                <xsl:value-of select="."/>
                        </field>
                        <xsl:text>&#xa;</xsl:text>
                        </xsl:for-each>
                </doc>
                </xsl:for-each>
        </add>
        </exsl:document>
</xsl:template>
share|improve this question
add comment

1 Answer

up vote 1 down vote accepted

Well, it seems there is a streaming XML option that's a bit different from XSLT (xslt requires the whole document to fit into memory, the streaming xml transformation languages don't).

Instead of re-writing the XSLT, which I had just spent a day preparing, I just spun up an AWS spot instance with 64 GB of ram, gave it some swap, and my 13GB XML consumed just about 65GB of ram under xsltproc.

On this system it ran, but I can now tell you that you won't get much more than a 13GB file out of even Amazon's largest high-memory instance. You'll need to go with another solution such as streaming XML transformations.

As a benchmark for anyone thinking of pushing the envelope. This transformation failed on an instance with 32 GB of RAM and 120G swap partition. It seems like you can swap a few gigs, but not too much before you hit some problem that will crash it.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.