Take the 2-minute tour ×
Ask Ubuntu is a question and answer site for Ubuntu users and developers. It's 100% free, no registration required.

I've got a XML file with 1GB (about 6000000 Lines). The goal is to split this file into smaller .xml files with more or less 20000 lines each.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE ICECAT-interface SYSTEM "http://data.icecat.biz/dtd/files.index.dtd">
<!-- source: Icecat.biz 2012 -->

<ICECAT-interface xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://data.icecat.biz/xsd/files.index.xsd">
  <files.index Generated="20120418012252">

    <file path="export/level4/PT/726463.xml" Product_ID="726463" Updated="20110815090811" Quality="ICECAT" Supplier_id="393" Prod_ID="174174" Catid="575" On_Market="0" Model_Name="Caplio R4 6.0MPix Silver" Product_View="4052" HighPic="http://images.icecat.biz/img/norm/high/726463-490.jpg" HighPicSize="102843" HighPicWidth="800" HighPicHeight="600" Date_Added="20070219000000">
      <EAN_UPCS>
        <EAN_UPC Value="4961311027048" />
      </EAN_UPCS>
    </file>

    ...

    <file path="export/level4/PT/738860.xml" Product_ID="738860" Updated="20101217151007" Quality="ICECAT" Supplier_id="169" Prod_ID="HEJ423020F9AT0C" Catid="219" On_Market="0" Model_Name="Endurastar J4K30" Product_View="1160" HighPic="http://images.icecat.biz/img/norm/high/738860-3355.jpg" HighPicSize="52197" HighPicWidth="175" HighPicHeight="212" Date_Added="20070219000000"></file>
  </files.index>
</ICECAT-interface>

For similar situations, I've used split or grep, but to this particular case, it gets trickier:

1)

The splitting process must occur under a specific node name, e.g, </file> to preserve the integrity of the file.

2)

The destination files must contain a header and a footer, to make the file valid and properly readable.

HEADER

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE ICECAT-interface SYSTEM "http://data.icecat.biz/dtd/files.index.dtd">
<!-- source: Icecat.biz 2012 -->

<ICECAT-interface xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://data.icecat.biz/xsd/files.index.xsd">
  <files.index Generated="20120418012252">

FOOTER

  </files.index>
</ICECAT-interface>

3)

Finally, my lack of solutions to this issue is due to the fact that this "action" will be repeated from time to time on a scheduled cron.

Note: The destination files can have any name, the directory lets assume to be the same where the script is running.

share|improve this question
    
Consider writing a ruby or python script for this, they have excellent support for xml. –  nikhil Jun 7 '12 at 10:29

1 Answer 1

One solution is to use Saxon-EE which supports streaming capability. For example

<xsl:stylesheet version="3.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">

<xsl:mode streamable="yes" on-no-match="shallow-copy"/>

<xsl:template match="file">
<xsl:result-document href="??????" 
  doctype-system="http://data.icecat.biz/dtd/files.index.dtd">

<ICECAT-interface xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"   
   xsi:noNamespaceSchemaLocation="http://data.icecat.biz/xsd/files.index.xsd">
  <files.index Generated="20120418012252">
   <xsl:copy-of select="."/>
  </files.index>
</ICECAT-interface>
</xsl:template>
share|improve this answer
    
Seems promising, but can't use it! By the documentation, the open source version runs under Java platform or .NET platform! –  Zuul Jun 7 '12 at 12:12
    
@Zuul Why would that keep you from using it? Both Java and (via Mono) .NET are available natively on Ubuntu. –  Eliah Kagan Jul 2 '12 at 10:40
    
@EliahKagan I do enjoy learning new stuff every day, but, that's not my area of expertise, so I was searching for a solution more related to my current knowledge. :) But hey, I'll give it a go nonetheless! –  Zuul Jul 2 '12 at 10:56

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.