Convert HTML to text format

Question

I want to extract the page content from this HTML file:

<BR />
<TABLE style=border-color:#32506d border=1 cellspacing=0>
    <caption class=header style=background-color:#32506d><b>Additional M2Ms & Standalone DataMasking List for 09 10 2020
            PST</b></caption>
    <tr style=background-color:#32506d class=header>
        <td class=CR>Start Time</td>
        <td class=CR>FM CR</td>
        <td class=CR>CR Type</td>
        <td class=CR>Customer Name</td>
        <td class=CR>Source Pod</td>
        <td class=CR>Target Pod</td>
        <td class=CR>DM Flag</td>
        <td class=CR>Release</td>
        <td class=CR>Data Center</td>
        <td class=CR>CDB Sync</td>
        <td class=CR>FreeSpace Check</td>
        <td class=CR>TDE/DV Check</td>
        <td class=CR>M2M Optin</td>
        <td class=CR>M2M Type</td>
        <td class=CR colspan=2>Database Reorg Details</td>
        <td>Operations Team</td>
    </tr>
    <tr>
        <td>09/10/2020-19:00</td>
        <td class=CR><a href=http://fleetmanager.oraclecloud.com/change/faces/registerChangeRequest?CRID=11124482
                target=_blank>11124482</td>
        <td>M2M</td>
        <td>TCS</td>
        <td>KCLB-CDB</td>
        <td>EGLG-TEST</td>
        <td class=CR>N</td>
        <td>Revision 13.20.07</td>
        <td>ks8-US-OCC</td>
        <td class=CR>
            <font color=#34A853>Yes</font>
        </td>
        <td class=CR>
            <font color=#34A853>Passed</font>
        </td>
        <td class=CR>
            <font color=#34A853>Passed</font>
        </td>
        <td class=CR>Y</td>
        <td class=CR>
            <font color=#34A853>sDC</font>
        </td>
        <td>
            <font color=#db3236>Reclaimable Space: 3532 GB</font>
        </td>
        <td>
            <font color=#db3236>Reorg Required</font>
        </td>
        <td>
            <center>
                <font color=#0000FF>RAMU</font>
            </center>
        </td>
    </tr>
    <tr>
        <td>09/10/2020-19:00</td>
        <td class=CR><a href=http://fleetmanager.oraclecloud.com/change/faces/registerChangeRequest?CRID=11170981
                target=_blank>11170981</td>
        <td>
            <font color=green>Standalone Data Masking</font>
        </td>
        <td>Wipro, Inc.</td>
        <td></td>
        <td>LMNO-TEST</td>
        <td class=CR></td>
        <td>Revision 13.20.07</td>
        <td>ns2-US</td>
        <td class=CR>NA</td>
        <td class=CR>NA</td>
        <td class=CR>NA</td>
        <td class=CR>NA</td>
        <td class=CR>NA</td>
        <td>
            <center>NA</center>
        </td>
        <td>
            <center>NA</center>
        </td>
        <td>DataMasking</td>
    </tr>
</TABLE><br /><span>Thanks,</span><br /><span>M2M Ops</span><br /><br /><span>Note: This is a system generated email,
    still you can reply with queries/suggestions.</span>

</HTML>

So far, I have tried doing so using sed:

sed -n '/^$/!{s/<[^>]*>//g;p;}' file.html

I am getting below output:

Start TimeFM CRCR TypeCustomer NameSource PodTarget PodDM FlagReleaseData CenterCDB SyncFreeSpace CheckTDE/DV CheckM2M OptinM2M TypeDatabase Reorg DetailsOperations Team
09/10/2020-19:0011124482
M2MTCSKCLB-CDBEGLG-TESTNRevision 13.20.07ks8-US-OCCYes
PassedPassedYsDCReclaimable Space: 3532 GBReorg RequiredRAMU
09/10/2020-19:0011170981
Standalone Data MaskingWipro Inc.LMNO-TESTRevision 13.20.07ns2-USNA
NANANANANANADataMasking
Thanks,M2M OpsNote: This is a system generated email, still you can reply with queries/suggestions.

However it is different from the desired output:

StartTime           FMCR      CRType                   CustomerName         SourcePod  TargetPod DMFlag Release               DataCenter       CDBSync  FreeSpaceCheck TDE/DVCheck M2MOptin  M2MType DatabaseReorgDetails                             OperationsTeam
09/10/2020-19:00    11124482    M2M                        TCS               KCLB-CDB  KCLB-TEST  N     Revision 13.20.07      ks8-US-OCC       YES     Passed          Passed      Y         sDC     Reclaimable Space: 3532 GB   Reorg Required     RAMU
09/10/2020-19:00    11170981 Standalone Data Masking     Wipro, Inc              LMNO-TEST              Revision 13.20.07      ns2-US           NA      NA               NA          NA         NA      NA                           NA              DataMasking

Do you specifically want to achieve the same with sed or awk? With sed and awk the output is completely dependent on the formatting of the HTML file. For example, since I have fixed formatting for the HTML file in your question, the output would change drastically. — Kulfy, Oct 31, 2020 at 9:05
How about using specialised tools such as html2text? html2text would provide the output like this paste which can further be modified using sed and other command line tools to get the desired output such as this another paste. Of course you can modify further. Also note the indentation and the position of elements depends on the HTML source and the width of the screen (in characters). Width can be supplied in html2text. — Kulfy, Oct 31, 2020 at 9:05
Try to open your html-document in a browser and then select all (Ctrl A), copy and paste the result in a text-editor — Bernard Decock, Oct 31, 2020 at 10:04
sed or awk anything should be fine need to convert html to text column wise — Srinivasulu Reddy, Oct 31, 2020 at 12:36

Kulfy · Accepted Answer · 2020-10-31 19:16:44Z

With sed and awk the output is completely dependent on the formatting of the HTML file. For example, the HTML source in revision #3 would yield different result when compared to that of revision #4.

Alternatively, you can use specific tools such as html2text. html2text would format the resultant HTML page into plain text characters. Of course you can further manipulate the output with other command-line tools such as sed and awk.

To install html2text, simply run:

sudo apt install html2text

To get started simply run:

html2text file.html

By default, html2text formats the HTML documents for a screen width of 79 characters. So, the result would look like this:


 ___________________________________Additional_M2Ms_&_Standalone_DataMasking_List_for_09_10_2020_PST____________________________________
|Start|FM CR   |CR Type   |Customer|Source|Target|DM  |Release |Data  |CDB |FreeSpace|TDE/DV|M2M  |M2M |Database Reorg      |Operations |
|Time_|________|__________|Name____|Pod___|Pod___|Flag|________|Center|Sync|Check____|Check_|Optin|Type|Details_____________|Team_______|
|09/  |        |          |        |      |      |    |        |      |    |         |      |     |    |Reclaimable|        |           |
|10/  |11124482|M2M       |TCS     |KCLB- |EGLG- |N   |Revision|ks8-  |Yes |Passed   |Passed|Y    |sDC |Space: 3532|Reorg   |   RAMU    |
|2020-|        |          |        |CDB   |TEST  |    |13.20.07|US-OCC|    |         |      |     |    |GB         |Required|           |
|19:00|________|__________|________|______|______|____|________|______|____|_________|______|_____|____|___________|________|___________|
|09/  |        |Standalone|        |      |      |    |        |      |    |         |      |     |    |           |        |           |
|10/  |11170981|Data      |Wipro,  |      |LMNO- |    |Revision|ns2-US|NA  |NA       |NA    |NA   |NA  |    NA     |   NA   |DataMasking|
|2020-|        |Masking   |Inc.    |      |TEST  |    |13.20.07|      |    |         |      |     |    |           |        |           |
|19:00|________|__________|________|______|______|____|________|______|____|_________|______|_____|____|___________|________|___________|

Thanks,
M2M Ops

Note: This is a system generated email, still you can reply with queries/
suggestions.

However, you can change width to the desired number of characters. For example, in your question the width is of 261 characters. Thus, you can also use

html2text -width 261 file.html

which would yield:


 _________________________________________________________________________________________________Additional_M2Ms_&_Standalone_DataMasking_List_for_09_10_2020_PST_________________________________________________________________________________________________
|Start_Time______|FM_CR___|CR_Type________________|Customer_Name|Source_Pod|Target_Pod|DM_Flag|Release__________|Data_Center|CDB_Sync|FreeSpace_Check|TDE/DV_Check|M2M_Optin|M2M_Type|Database_Reorg_Details____________________________________|Operations_Team___|
|09/10/2020-19:00|11124482|M2M____________________|TCS__________|KCLB-CDB__|EGLG-TEST_|N______|Revision_13.20.07|ks8-US-OCC_|Yes_____|Passed_________|Passed______|Y________|sDC_____|Reclaimable_Space:_3532_GB|Reorg_Required_________________|_______RAMU_______|
|09/10/2020-19:00|11170981|Standalone_Data_Masking|Wipro,_Inc.__|__________|LMNO-TEST_|_______|Revision_13.20.07|ns2-US_____|NA______|NA_____________|NA__________|NA_______|NA______|____________NA____________|______________NA_______________|DataMasking_______|

Thanks,
M2M Ops

Note: This is a system generated email, still you can reply with queries/suggestions.

Now, to manipulate things, for example, removing glyph(|), underscore(_), empty lines, the very first line and last 3 lines, you can use any command line tools as per your requirement. An ugly method would look like

html2text -width 200 file.html | sed 's/|/\ /g;s/\_/\ /g;/^$/d'| head -n -3 | tail -n +2

This would produce

 Start Time     FM CR    CR Type      Customer Name Source Pod Target Pod DM Flag Release  Data Center CDB Sync FreeSpace TDE/DV Check M2M Optin M2M Type Database Reorg Details        Operations Team 
                                                                                                                Check                                                                                   
 09/10/2020-19: 11124482 M2M          TCS           KCLB-CDB   EGLG-TEST  N       Revision ks8-US-OCC  Yes      Passed    Passed       Y         sDC      Reclaimable    Reorg Required      RAMU       
 00                                                                               13.20.07                                                                Space: 3532 GB                                
 09/10/2020-19: 11170981 Standalone   Wipro, Inc.              LMNO-TEST          Revision ns2-US      NA       NA        NA           NA        NA             NA             NA       DataMasking     
 00                      Data Masking                                             13.20.07

we can not install the html2text tool ,need to do sed or awk — Srinivasulu Reddy, Nov 1, 2020 at 16:35
@SrinivasuluReddy Did you encounter any error while installing? — Kulfy, Nov 1, 2020 at 16:42
@yurenchen Well there's -utf8 option available with html2text. You can try that. — Kulfy, Dec 16, 2020 at 12:39
@Kulfy thanks, I known this option, tried. ubuntu 18 pkg and the last src in github. // seems not work well — yurenchen, Dec 16, 2020 at 16:24

Jose Pepe Flores Peters · Accepted Answer · 2022-01-11 04:57:06Z

I prefer use "vim" with "sed sentences" in a file-command with instructions, then execute with "vim -s file-command".

$ cat file-command
:%join
:%s//&\r/gi
:%s//\t/gi
:%s/]*>//g
:w %.txt
:q!


$ vim -s file-command example.html

$ cat example.html.txt
 Additional M2Ms & Standalone DataMasking List for 09 10 2020 PST 
 Start Time  FM CR   CR Type     Customer Name   Source Pod  Target Pod  DM Flag     Release     Data Center     CDB Sync    FreeSpace Check     TDE/DV Check    M2M Optin   M2M Type    Database Reorg Details  Operations Team      
 09/10/2020-19:00    11124482    M2M     TCS     KCLB-CDB    EGLG-TEST   N   Revision 13.20.07   ks8-US-OCC   Yes     Passed      Passed     Y    sDC     Reclaimable Space: 3532 GB      Reorg Required       RAMU       
 09/10/2020-19:00    11170981     Standalone Data Masking    Wipro, Inc.         LMNO-TEST       Revision 13.20.07   ns2-US  NA  NA  NA  NA  NA   NA      NA     DataMasking

Stack Exchange Network

Convert HTML to text format

2 Answers 2

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged
command-line
bash
sed
awk
.

Hot Network Questions

Convert HTML to text format

2 Answers 2

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged command-linebashsedawk.

Related

Hot Network Questions

Not the answer you're looking for? Browse other questions tagged
command-line
bash
sed
awk
.