1

System

Linux local 5.0.0-27-lowlatency #28-Ubuntu SMP PREEMPT Tue Aug 20 20:33:37 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Question

How can I compare two cols divided by , and fields in " when , is present in field (classic csv with ").

Input file.

Titulka,231096,Spárovka borová 18 x 200 x 600 mm,https://eshop.unihobby.cz/stavba-drevo-a-wpc-sparovky-sparovka-borova-18x200x600mm/144020p/?utm_source=kupi&utm_medium=letak&utm_campaign=9_2019,https://eshop.unihobby.cz/bin/product/4/231096-sp-rovka-borov-1-pou-t-tuto.jpg,42,45
Titulka,345602,"Lak Aquafin Plus čirý lesklý 0,6 kg",https://eshop.unihobby.cz/bydleni-barvy-a-laky-laky-a-lazury-lak-aquafin-plus-ciry-leskly-0-6-kg/153252p/?utm_source=kupi&utm_medium=letak&utm_campaign=9_2019,https://eshop.unihobby.cz/bin/product/4/345602-aquafin-plus-plechovka.jpg,179,199
Titulka,345588,"Lak Movi Pur OSB čirý lesklý 0,6 kg",https://eshop.unihobby.cz/bydleni-barvy-a-laky-laky-a-lazury-lak-movi-pur-osb-ciry-leskly-0-6-kg/150470p/?utm_source=kupi&utm_medium=letak&utm_campaign=9_2019,https://eshop.unihobby.cz/bin/product/4/345588.jpg,209,229
Titulka,213993,Schodnice masiv 300x1000x40 mm - smrk,https://eshop.unihobby.cz/stavba-schodiste-a-zabradli-schodnice-schodnice-masiv-300x1000x40-mm-smrk/138451p/?utm_source=kupi&utm_medium=letak&utm_campaign=9_2019,https://eshop.unihobby.cz/bin/product/4/213993-schodnice-masiv-300x1000x40-smrk.jpg,299,329
2,233292,Střešní okno plastové kyvné Klasik NK E1 - 55 x 78 cm,https://eshop.unihobby.cz/stavba-okna-stresni-okna-stresni-okno-plastove-kyvne-klasik-nk-e1-55-x-78-cm/144206p/?utm_source=kupi&utm_medium=letak&utm_campaign=9_2019,https://eshop.unihobby.cz/bin/product/4/233292-2-.jpg,3990,3990
2,234185,Střešní okno dřevěné kyvné Klasik MNK S1 - 55 x 78 cm,https://eshop.unihobby.cz/stavba-okna-stresni-okna-stresni-okna-stresni-okno-drevene-kyvne-klasik-mnk-s1-55-x-78-cm/144478p/?utm_source=kupi&utm_medium=letak&utm_campaign=9_2019,https://eshop.unihobby.cz/bin/product/4/233288-3--4.jpg,2990,2990

...

Expected output file (compare $6 and $7, write $7 when $6 != $7).

Titulka,231096,Spárovka borová 18 x 200 x 600 mm,https://eshop.unihobby.cz/stavba-drevo-a-wpc-sparovky-sparovka-borova-18x200x600mm/144020p/?utm_source=kupi&utm_medium=letak&utm_campaign=9_2019,https://eshop.unihobby.cz/bin/product/4/231096-sp-rovka-borov-1-pou-t-tuto.jpg,42,45
Titulka,345602,"Lak Aquafin Plus čirý lesklý 0,6 kg",https://eshop.unihobby.cz/bydleni-barvy-a-laky-laky-a-lazury-lak-aquafin-plus-ciry-leskly-0-6-kg/153252p/?utm_source=kupi&utm_medium=letak&utm_campaign=9_2019,https://eshop.unihobby.cz/bin/product/4/345602-aquafin-plus-plechovka.jpg,179,199
Titulka,345588,"Lak Movi Pur OSB čirý lesklý 0,6 kg",https://eshop.unihobby.cz/bydleni-barvy-a-laky-laky-a-lazury-lak-movi-pur-osb-ciry-leskly-0-6-kg/150470p/?utm_source=kupi&utm_medium=letak&utm_campaign=9_2019,https://eshop.unihobby.cz/bin/product/4/345588.jpg,209,229
Titulka,213993,Schodnice masiv 300x1000x40 mm - smrk,https://eshop.unihobby.cz/stavba-schodiste-a-zabradli-schodnice-schodnice-masiv-300x1000x40-mm-smrk/138451p/?utm_source=kupi&utm_medium=letak&utm_campaign=9_2019,https://eshop.unihobby.cz/bin/product/4/213993-schodnice-masiv-300x1000x40-smrk.jpg,299,329
2,233292,Střešní okno plastové kyvné Klasik NK E1 - 55 x 78 cm,https://eshop.unihobby.cz/stavba-okna-stresni-okna-stresni-okno-plastove-kyvne-klasik-nk-e1-55-x-78-cm/144206p/?utm_source=kupi&utm_medium=letak&utm_campaign=9_2019,https://eshop.unihobby.cz/bin/product/4/233292-2-.jpg,3990 /* here is $6 = $7, so no output for $7 */
2,234185,Střešní okno dřevěné kyvné Klasik MNK S1 - 55 x 78 cm,https://eshop.unihobby.cz/stavba-okna-stresni-okna-stresni-okna-stresni-okno-drevene-kyvne-klasik-mnk-s1-55-x-78-cm/144478p/?utm_source=kupi&utm_medium=letak&utm_campaign=9_2019,https://eshop.unihobby.cz/bin/product/4/233288-3--4.jpg,2990 /* here is $6 = $7, so no output for $7 */

...

I tried

awk -F ',' '{ if ($6 != $7) print $7; }' FILE

Returns bad values and return only $7 field.

Thanks.

Improve this question
2
  • Im not sur to well understand the question, please try awk -F',' '$6!=$7 {print $7}' and tell me if its ok for you, ill make it an answer
    – cmak.fr
    Sep 11, 2019 at 7:54
  • @cmak.fr Hi, thanks, but this is the same. Problem is that some fields are in doublequotas, " field" as one field, but awk takes , independently on this. For example 345588,"Lak Movi Pur OSB čirý lesklý 0,6 kg", has , inside ""and it should be take as one field for awk. But awk separate it by ,. And how get all values. Simple said, I need just remove last value, if the value is the same as before, for example last 2 fields ...,12,12 ar the same, so return only ...,12, ...,12,14 are different, so return ...,12,14.
    – genderbee
    Sep 11, 2019 at 8:03

1 Answer 1

Reset to default
1

If you have a CSV with commans inside fields, awk is not the best solution.
For your example file, you could still use it, because you can simply compare the last two fields and don't care what is there before:

awk '
BEGIN{FS=OFS=","}
$NF==$(NF-1){$NF=""}1
' file

But in general, it would be better parsing CSV with a proper CSV parser, e.g. csvkit:

csvsql -H --query '
    SELECT a,b,c,d,e,f, case g when f then NULL else g END g from file
' file | tail -n+2

(I added tail -n+2 to suppress the header row)

Note that both solutions will keep the total number of fields, just emptying the last field. To remove the last comma, add | sed 's/,$//'.

Improve this answer
5
  • Thanks, quite simple good solution, just there is , at the end of line, only last value $7 is removed. How can I remove , too in this case? And I can't use csvsql, because I don't have root access to install it.
    – genderbee
    Sep 11, 2019 at 9:13
  • if you read the full answer ....
    – pLumo
    Sep 11, 2019 at 9:13
  • Sorry, thanks. But in case need to do it for fields in the middle of line, is solution for awk for parsing line with doublequotas ""`? Thanks.
    – genderbee
    Sep 11, 2019 at 9:26
  • The solution would be to ask your admin to install proper tools to fulfill your job. csvkit or some python csv module would be good. There won't be any good awk or sed solution for that. Maybe some weird error-prone workaround that works somehow for some cases, but no real solution.
    – pLumo
    Sep 11, 2019 at 9:32
  • At least with gawk and mawk, you can decrement NF to remove the last field without leaving a trailing delimiter
    – steeldriver
    Sep 11, 2019 at 12:02

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .