Questions tagged [text-processing]
Questions about modifying, creating or parsing plain text files. This is a very broad tag, if your question is about using a specific tool, please tag it with the tool's name as well (e.g. sed, awk, perl, etc.)
1,126
questions
0
votes
1
answer
22
views
Compare two files and remove every line that doesn't match with file B
I have two files. file A:
Modern Tales: Age Of Invention Playstation 4, 1935, 3478-3480
Modern Tales: Age Of Invention Xbox One, 3074
Moero Chronicle Switch, 6667, 12400, 28910, 29900, 29901, 29920
...
0
votes
2
answers
68
views
script to compare two files and remove match line in file A
I have two files.
file A:
Modern Tales: Age Of Invention Playstation 4, 1935, 3478-3480
Modern Tales: Age Of Invention Xbox One, 3074
Moero Chronicle Switch, 6667, 12400, 28910, 29900, 29901, 29920
...
-3
votes
1
answer
45
views
cut even lines and merge to upper line
I have a file like with CRLF line terminators:
Green Day: Rock Band Xbox 360
3074
Green Hell Steam
27015-27030, 27036-27037
Green Hell Switch
6667, 12400, 28910, 29900, 29901, 29920
Adios Steam
Adios ...
1
vote
3
answers
44
views
extract colname
I need to get colnames but just want one colname in each line so could you please help how to change below command.
head -n 1 betas_1.csv >col.names
- 137
2
votes
2
answers
161
views
bash script to append file's name at the end of each line in it's file
I have some files in a directory:
a.csv b.csv c.csv
I wanted to write a bash script to add a filename at the end of each line of its file, like:
a.csv
line1 ,a
line2, a
line, a
b.csv
line1, b
line2, ...
1
vote
1
answer
91
views
appending variable to end of file with sed
I have dates on the last line in files like this:
20230220
and I want to add hours like this:
2023022000
The hours are continuously assigned to a variable i in a loop.
How to use sed inside the loop ...
- 11
1
vote
1
answer
107
views
grep and awk commands not returning or storing output to the variable
I have a (users.csv) file containing a list of id's of all employees. I need is to loop through this file for each employee, search if it exists in backup.txt file using grep and store the output in a ...
-1
votes
1
answer
32
views
Trying to remove all text except for IPs, data is in a file I downloaded
This is what I normally use to normalize the data, but it is not working. I need to keep only the IPs, one IP per line, and everything else must go. I'm using this in an .sh shell script.
grep -oE '[...
- 1
0
votes
0
answers
12
views
How to match results with grep from a list of words in a file? [duplicate]
for this example:
grep 'word1\|word2\|word3' /path/to/file
can these be placed in a file?
something like:
grep '<file_with_a_list_of_words>' /path/to/file
for a large list of words to match.
1
vote
2
answers
317
views
awk or sed command to replace line break plus text containing spaces
An answer to another question suggests sed -i 's/original/replacement/g' file.txt to replace specific words in a text file. My starting situation looks like this:
Item: PRF
Type: File
...
- 313
0
votes
0
answers
76
views
Tail in combination with sort
I use tail in a variable to echo contents of multiple files in a folder.
VAR="$(tail -n +1 DIR/*/file)"
This outputs all the files contents with the needed filename on top like
==> /...
- 111
6
votes
1
answer
361
views
How to sort by year-month-day from ascending using the "sort" command?
I want to sort dates in the format 31.12.2023. Lowest year should be displayed first then lowest month then lowest day.
Is the following code fitting my needs? Seems good on first look.
sort -t '.' -k ...
- 111
-4
votes
1
answer
147
views
count frequency values of all csv
I have a large csv file (without header or index), for example
A T C G
G T A C
CT T A G
G G G G
I want to count all values in all csv (not for special column or row), the output will be:
A 3
T 3
C 2
...
0
votes
6
answers
158
views
Cut last two lines of a file and append to another file repeatedly
I tried the following command to cut out the last two lines of a file and append them to a new file
tail -n -2 file1 >> file2 && head -n -2 file1 > tmp && mv tmp file1
With ...
- 111
-1
votes
2
answers
159
views
Remove last occurrence in a file with sed
I want to remove the very last occurrence of "(* cds_ams_schematic *)" in a file using sed command.
`view schematic
`timescale 1ns / 1ns
(* cds_ams_schematic *)
module ...
endmodule
// ...
2
votes
1
answer
165
views
Grep only until certain symbol
I try to grep some string and want grep to stop at a '+' symbol. The searched string could be "foo" or "foo+bar". Grep treats all findings no matter what comes after foo and the -w ...
- 111
0
votes
1
answer
371
views
How to use "sed" for replacing xml tag in multiple lines xml content on bash
I want to use sed to replace the XML tag on bash.
Here is the example xml content for testing:
<xml-content>
<validation>
<timeout>2880</timeout>
<subject>example</...
- 179
2
votes
1
answer
41
views
How to search output of help message of program (using e.g. grep)
I have a directory with executables and I want to search for a word in the output of the help messages of these executables, i.e. the text that is printed in the console after using the command ./...
- 497
0
votes
1
answer
107
views
Using awk to parse for conditions over multiple lines? (ifconfig output)
I'm trying to parse ifconfig output with awk.
Right now my script is:
# Setup
ifconfig > ip.txt
ans=$(cat ip.txt | awk '$1 == "inet" {print $2}')
#output
echo "Here are your IPs:&...
- 1
0
votes
2
answers
104
views
Ubuntu Bash - print specific word from a file
I need some help with some bash. I've only learnt like grep, awk, find, sed etc.
Say I have a file named people.txt with different names on a line on multiple lines but a name appears in every line. ...
0
votes
1
answer
195
views
Can globbing be used to search file contents?
I basically understand that globbing is used for filename expansion in the shell. I also understand that regular expressions are used for text pattern matching within a file. However, is globbing also ...
- 129
1
vote
2
answers
838
views
How to delete a range of lines in a text file?
I want to know how to quickly delete text with Bash. For example, I have a file named file.txt with a total of 500 lines.
I want to delete lines 80 to 146 - how do I do that?
What's is a terminal ...
- 303
1
vote
2
answers
51
views
How to monitor a growing log file for specific string and print the line to terminal in realtime?
For example. I have nohup.out.log file which is growing over time. I want to monitor this file and check if the string test error occurs in this file and print the corresponding lines to terminal or ...
- 11
4
votes
4
answers
2k
views
Convert text file to CSV on shell
I would like to convert a text file into CSV format.
Here is an excerpt from the file:
{"Outdated":false,"Watt":233,"Timestamp":1669647142,"A_Plus":6523.896,&...
- 51
0
votes
0
answers
17
views
How to deal with large formatted documents quickly in Ubuntu?
I have a .docx file that have a size of 200MB.
To open or save such huge files, MS Office on Windows 10 only need several seconds, but Libre Office on Ubuntu 22.04 take dozens of seconds.
(FYI: iWork ...
- 283
2
votes
2
answers
1k
views
Replace a line of text in a file, with the contents of another file
I have two files, let's call them foo.py and fixed_rule_map. I need to:
find the line in foo.py that contains the pattern rule_map = . It would be enough to find the pattern rule_map, if that's ...
- 285
2
votes
1
answer
242
views
How to log URL to a file only if it’s HTTP status code is 200?
I want to make a script so I can check if I can connect to several URL.
For example I do curl -i -L --silent URL | grep "HTTP/2 200" >> *.txt
I want to combine it with an if statement ...
- 21
2
votes
1
answer
210
views
Oneliner to get the command which started a process on a certain port
I would need a one-line command which, given a port number, returns information on the command which started the process running on that port.
Usually, I get this information by running a sudo netstat
...
- 195
5
votes
2
answers
890
views
How do I replace multiple fields in multiple XML files?
I have around 4000 XML files and I need to replace the value of both <filename> and <path> fields.
I need to replace those fields dynamically. e.g. images0001.xml should have images0001 ...
0
votes
3
answers
335
views
How to select several words, and remove all lines that contain those words?
See screenshot. I want to get rid of the clutter, and leave only things like ID, and TEXT, but remove the other things, such as type, date, date_unixtime, edited, from, etc.
I tried the following ...
user1613265
0
votes
1
answer
43
views
Extract [a-c] from file and save in another file
I have a file with alphabetically sorted lines. The file starts from 0-9 till a-z. Example
000dasd
001dss
awer
addd
alko
babb
bobb
...
zzasd
Since the file is really big I want to divide it into ...
- 3
1
vote
3
answers
231
views
To make separate files based on pattern within a single file
I have a file containing 2.3M lines. Which looks like:
$less V2.fastq
>TS19_EWP4IQK02JPFP5
CATGCTGCCTCCCGTAGGAGTTTGGTCCGTGTCTCAGTACCAATGTGGGGGACCTTCCTC
...
- 59
-1
votes
2
answers
27
views
How to fail command if zero or more than one line in output
Let's say I have a command like
git log --some-options
and I expect it to produce exactly one line so I could feed it to xargs, like git log --some-options | xargs git branch --all --contains
how do I ...
- 191
0
votes
2
answers
43
views
How do you break up lines character-wise?
I have a big text file made up of 64 hex values.
Each line is stored one after the other; some lines are incorrectly recorded with 63 character hex values, while others are stored with 61 character ...
- 91
2
votes
4
answers
1k
views
How to print the content from a text file with one blank line after each line?
This is what I type and the output I get:
$ cat helloworld.txt
Hello World!
I'm Kanladaporn Sirithatthamrong
6480952
But this is what I want an output to be:
Hello World!
I'm Kanladaporn ...
user1637069
0
votes
2
answers
58
views
sed / awk / etc - Sorting merged text before paste OR paste special: new line
My scenario:
On an internal platform a list of IPs is displayed like:
8.8.8.8 1.0.0.1 1.1.1.1 192.168.1.1 192.168.0.1
When I copy these values and put them (pasted) in a text file they are all pasted, ...
- 163
0
votes
0
answers
29
views
script for heap utilization on ColdFusion server
I have a command which return the output for heap size utilization:-
jcmd `jps -l | grep com.adobe.coldfusion.bootstrap.Bootstrap | cut
-f1 -d ' '` GC.heap_info
it gives output like this
16144:
...
3
votes
2
answers
4k
views
How can I delete specific string from a file
I do have a bash script. I want to delete a specific string from a text file. I already found a command on the online but it seems it isn't that stable..
I've tried;
mv .admins .adminsold
sed -r "...
- 379
6
votes
2
answers
8k
views
How can I check if string exists in file
I'm currently writing a bash script that should check if the exact string 329, exists in myfile. I have searched through the web and found some answers, but I can't use -x parameters because I have ...
- 379
3
votes
3
answers
1k
views
How to insert an ascii character in a line position?
I would like to edit a list of MACs in bl:ab:la:bl:ab:la format >> in to blab.labl.abla format. Each MAC is in a line.
I'm new in the use of AWK or SED but I'm almost sure that it can be ...
- 113
0
votes
1
answer
209
views
grep command to get file list with concatenated custom text
I am trying to list of files' names with custom text(my another command) with it. I am using below command:
grep -rl --include=*.php --include=*.html --include=*.js 'ajax.googleapis.com/ajax/libs/...
- 219
0
votes
0
answers
93
views
PDF Post Processing with Evince (Document Viewer)
I have a scanned PDF document which I've opened with Ubuntu's document viewer (Evince). When I select a part of the text in the document, it automatically processes the scanned text and converts it to ...
2
votes
4
answers
788
views
how to separate each content in the file?
I have a text file (more then 1GB in size) and it contains lines like these:
...
- 91
0
votes
2
answers
452
views
How to remove characters/strings/patterns between, before and after some other characters/strings/patterns in a fasta file using a Ubuntu command line
How can I change all the lines in a fasta file from this:
>vsearch_derep1;size=1 331 95 544 TRINITY_DN40607_c0_g1_i1 len=2000path=[0:0-1097]
...
- 13
0
votes
2
answers
353
views
Sed (or other command) replace character range with hex codes
Let's say we have a file with non-printable characters.
I would like to visualize those using their hex codes. Arte there a single liners that can achieve it, without writing a full-blown program.
say ...
- 283
0
votes
1
answer
180
views
How can I extract text between lines containing timestamps?
I'm looking to text process an existing file and have the output of the alrorithm output to a new file. I thought it would be an easy task, but it's flummoxing me, probably because I don't know ls ...
- 9
0
votes
0
answers
204
views
How do I get formatted responses from terminal?
I'm struggling to understand the response that I receive from the terminal as it is all pasted in one line without indentation, line breaks, etc. Is there a way to enable that? Or some command? This ...
0
votes
2
answers
124
views
from single column file to multi column file
I have a file like this:
azg
1234567890
asgz
1234367890
aztyu
1234377890
ast
1234367899
asgassfa
1134367890
and I want to create an output to look like this:
did dialer-label
1234567890 ...
- 11
0
votes
0
answers
137
views
XML content replace
I Have json example of replacing the json content dynamically as below and printing the id after creating an issue in jira .. Same operation needs to be done for TestNGXML report
Kindly suggest the ...
- 1
0
votes
1
answer
1k
views
Parse JSON in Bash
I'm making a polyglot program, which takes input and runs files. I would like to run a command for a list of languages. Currently, I only have JavaScript and Ruby supported. My JSON file named ...
- 133