Take the 2-minute tour ×
Ask Ubuntu is a question and answer site for Ubuntu users and developers. It's 100% free, no registration required.

I want to form a network connection to a website and read the data line by line and store it in a text file in my system using shell scripting. I have done it using java where I can read using a URLConnection object to that particular resource.
In shell scripting is WGET Spider the only way to do it? If not what other ways are there to read text file from a website, parse it and store it in my local directory?

EDIT

I tried using WGET using wget -o /home/user/Desktop/training.txt https://www.someurl.com. But the output is something like this

--2014-04-15 00:39:15--  https://s3.amazonaws.com/hr-testcases/368/assets/trainingdata.txt
Resolving s3.amazonaws.com (s3.amazonaws.com)... 176.32.99.154
Connecting to s3.amazonaws.com (s3.amazonaws.com)|176.32.99.154|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1554016 (1.5M) [text/plain]
Saving to: ‘trainingdata.txt.1’

     0K .......... .......... .......... .......... ..........  3% 47.5K 31s
    50K .......... .......... .......... .......... ..........  6%  129K 20s
   100K .......... .......... .......... .......... ..........  9%  136K 16s
   150K .......... .......... .......... .......... .......... 13%  149K 14s
   200K .......... .......... .......... .......... .......... 16% 1.57M 11s
   250K .......... .......... .......... .......... .......... 19%  162K 10s
   300K .......... .......... .......... .......... .......... 23%  678K 9s
   350K .......... .......... .......... .......... .......... 26%  612K 7s
   400K .......... .......... .......... .......... .......... 29%  307K 7s
   450K .......... .......... .......... .......... .......... 32%  630K 6s
   500K .......... .......... .......... .......... .......... 36%  699K 5s
   550K .......... .......... .......... .......... .......... 39%  520K 5s
   600K .......... .......... .......... .......... .......... 42%  580K 4s
   650K .......... .......... .......... .......... .......... 46%  516K 4s
   700K .......... .......... .......... .......... .......... 49%  551K 3s
   750K .......... .......... .......... .......... .......... 52%  713K 3s
   800K .......... .......... .......... .......... .......... 56%  720K 3s
   850K .......... .......... .......... .......... .......... 59%  701K 2s
   900K .......... .......... .......... .......... .......... 62%  603K 2s
   950K .......... .......... .......... .......... .......... 65%  670K 2s
  1000K .......... .......... .......... .......... .......... 69%  715K 2s
  1050K .......... .......... .......... .......... .......... 72%  671K 1s
  1100K .......... .......... .......... .......... .......... 75%  752K 1s
  1150K .......... .......... .......... .......... .......... 79%  535K 1s
  1200K .......... .......... .......... .......... .......... 82%  607K 1s
  1250K .......... .......... .......... .......... .......... 85%  675K 1s
  1300K .......... .......... .......... .......... .......... 88%  727K 1s
  1350K .......... .......... .......... .......... .......... 92%  707K 0s
  1400K .......... .......... .......... .......... .......... 95%  632K 0s
  1450K .......... .......... .......... .......... .......... 98%  785K 0s
  1500K .......... .......                                    100%  931K=4.5s

2014-04-15 00:39:23 (341 KB/s) - ‘trainingdata.txt.1’ saved [1554016/1554016]

It only seems to give the statistics like the time taken to download and all. It does not give save the actual data from the url.

share|improve this question
    
@terdon Oh ok. Can't I control the location of the downloaded file? –  Ashwin Apr 14 at 19:37
    
No, hang on, my bad. Writing an answer now. What you're saving is the standard error. –  terdon Apr 14 at 19:46

2 Answers 2

It sounds like you want netcat,

Netcat is a featured networking utility which reads and writes data across network connections, using the TCP/IP protocol. It is designed to be a reliable "back-end" tool that can be used directly or easily driven by other programs and scripts. At the same time, it is a feature-rich network debugging and exploration tool, since it can create almost any kind of connection you would need and has several interesting built-in capabilities.

for more, you can always man nc

share|improve this answer
    
how is netcat different form wget? –  Ashwin Apr 14 at 18:56
    
@Ashwin From man wget - GNU Wget is a free utility for non-interactive download of files from the Web. In contrast, netcat is for scripting network connections. –  Elliott Frisch Apr 14 at 18:57

The command you are running is using the -o flag which does (from man wget):

   -o logfile
   --output-file=logfile
       Log all messages to logfile.  The messages are normally reported to
       standard error.

It does not actually save the target of the URL to that file, only the standard error of wget. By default, wget saves the target with the same name as the remote file. Do, for example this

wget http://www.foo.com/index.html

will save the file as index.html in your current directory. To specify a different name for the file, use -O (CAPITAL o, as in Oliver) instead:

   -O file
   --output-document=file
       The documents will not be written to the appropriate files, but all
       will be concatenated together and written to file.  If - is used as
       file, documents will be printed to standard output, disabling link
       conversion.  (Use ./- to print to a file literally named -.)

       Use of -O is not intended to mean simply "use the name file instead
       of the one in the URL;" rather, it is analogous to shell
       redirection: wget -O file http://foo is intended to work like wget
       -O - http://foo > file; file will be truncated immediately, and all
       downloaded content will be written there.
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.