I've recently gotten into using tools like
cat, etc. because I have to deal with some very large CSV files (>10GB) which aren't quite delimited correctly (for instance, having occurrences of the delimiter character inside some of the fields.
In my working with one of these files, I've run the following command in the process of trying to figure out a way to correctly identify which instances of
; is a delimiter and replace them with some other character:
grep -v -n --text "[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9].*[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9].*[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9].*[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]" < Transactions.csv
The regex can probably be done much better, but anyway; what is surprising is that, among others, the above code outputs the following line:
12345678:2016-10-25;12345678912345;2016-10-25;gobbledegook �IDNR: 69 ;12345.67;.00;2003-09-05;12345678;2003-09-03;stuff stuff ;12345 fgadfkjgbsdkb;12/3/45678/9
(as this was actually transaction data, I've changed most of the fields' values, except for the offending
�) Maybe I'm being silly, but why doesn't the above regex match that line? It seems like the regex
.* somehow doesn't match that character for some reason.
I suspect that the file is saved using the UTF-16 encoding, if that makes any difference.
Edit: Thanks to @exore for the answer. As it turns out, my file was encoded in ISO-8859-15, which I was able to figure by
grepping out the lines containing special characters, which were relatively few, into a file and opening that in gedit. I then used
iconv to convert that to utf8, after which it worked fine!