0

I've got a XML file with 1GB (about 6000000 Lines). The goal is to split this file into smaller .xml files with more or less 20000 lines each.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE ICECAT-interface SYSTEM "http://data.icecat.biz/dtd/files.index.dtd">
<!-- source: Icecat.biz 2012 -->

<ICECAT-interface xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://data.icecat.biz/xsd/files.index.xsd">
  <files.index Generated="20120418012252">

    <file path="export/level4/PT/726463.xml" Product_ID="726463" Updated="20110815090811" Quality="ICECAT" Supplier_id="393" Prod_ID="174174" Catid="575" On_Market="0" Model_Name="Caplio R4 6.0MPix Silver" Product_View="4052" HighPic="http://images.icecat.biz/img/norm/high/726463-490.jpg" HighPicSize="102843" HighPicWidth="800" HighPicHeight="600" Date_Added="20070219000000">
      <EAN_UPCS>
        <EAN_UPC Value="4961311027048" />
      </EAN_UPCS>
    </file>

    ...

    <file path="export/level4/PT/738860.xml" Product_ID="738860" Updated="20101217151007" Quality="ICECAT" Supplier_id="169" Prod_ID="HEJ423020F9AT0C" Catid="219" On_Market="0" Model_Name="Endurastar J4K30" Product_View="1160" HighPic="http://images.icecat.biz/img/norm/high/738860-3355.jpg" HighPicSize="52197" HighPicWidth="175" HighPicHeight="212" Date_Added="20070219000000"></file>
  </files.index>
</ICECAT-interface>

For similar situations, I've used split or grep, but to this particular case, it gets trickier:

1)

The splitting process must occur under a specific node name, e.g, </file> to preserve the integrity of the file.

2)

The destination files must contain a header and a footer, to make the file valid and properly readable.

HEADER

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE ICECAT-interface SYSTEM "http://data.icecat.biz/dtd/files.index.dtd">
<!-- source: Icecat.biz 2012 -->

<ICECAT-interface xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://data.icecat.biz/xsd/files.index.xsd">
  <files.index Generated="20120418012252">

FOOTER

  </files.index>
</ICECAT-interface>

3)

Finally, my lack of solutions to this issue is due to the fact that this "action" will be repeated from time to time on a scheduled cron.

Note: The destination files can have any name, the directory lets assume to be the same where the script is running.

Improve this question
1
  • Consider writing a ruby or python script for this, they have excellent support for xml.
    – nikhil
    Jun 7, 2012 at 10:29

1 Answer 1

Reset to default
0

One solution is to use Saxon-EE which supports streaming capability. For example

<xsl:stylesheet version="3.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">

<xsl:mode streamable="yes" on-no-match="shallow-copy"/>

<xsl:template match="file">
<xsl:result-document href="??????" 
  doctype-system="http://data.icecat.biz/dtd/files.index.dtd">

<ICECAT-interface xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"   
   xsi:noNamespaceSchemaLocation="http://data.icecat.biz/xsd/files.index.xsd">
  <files.index Generated="20120418012252">
   <xsl:copy-of select="."/>
  </files.index>
</ICECAT-interface>
</xsl:template>
Improve this answer
3
  • Seems promising, but can't use it! By the documentation, the open source version runs under Java platform or .NET platform!
    – Zuul
    Jun 7, 2012 at 12:12
  • @Zuul Why would that keep you from using it? Both Java and (via Mono) .NET are available natively on Ubuntu.
    – Eliah Kagan
    Jul 2, 2012 at 10:40
  • @EliahKagan I do enjoy learning new stuff every day, but, that's not my area of expertise, so I was searching for a solution more related to my current knowledge. :) But hey, I'll give it a go nonetheless!
    – Zuul
    Jul 2, 2012 at 10:56

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .