0

I'm having some difficulty downloading a zip file from a server URL using wget and curl. Neither worked the way they're supposed to work. For example:

wget  [email protected] --password=secret!  "https://msi.domain.com/admin/ui/feedbackCSV/organizationReports/sjf3j45345345bsdf?reports[]=pageviews& reports[]=searches&from=Nov 12 2020&to=Nov 19 2020/client_Technical_Publications_stats.zip"

I got 400 response which is a bad request error , so I thought maybe I need to encode the URL.

I tried with:

wget  [email protected] --password=secret! 
"https%3A%2F%2Fmsi.domain.com%2Fadmin%2Fui%2FfeedbackCSV%2ForganizationReports%2Fsjf3j45345345bsdf%3Freports%5B%5D%3Dpageviews%26%20reports%5B%5D%3Dsearches%26from%3DNov%2012%202020%26to%3DNov%2019%202020%2Fclient_Technical_Publications_stats.zip"

After a few seconds it's creating a file name with 2Fsjf3j45345345bsdf?reports[]=pageviews& reports[]=searches&from=Nov 12 2020&to=Nov 19 2020%2Fclient_Technical_Publications_stats.zip and, when I open the file, I noticed bunch of HTML in it.

If copy and paste my encoded URL into the browser, I'm able to download the actual zip file.

I used the chrome developer tool to find the headers in the request  and I find below headers 
**Request  headers**
:authority: msi.domain.com
:method: GET
:path: /admin/ui/feedbackCSV/organizationReports/2Fsjf3j45345345bsdf?reports%5B%5D=pageviews&%20reports%5B%5D=searches&from=Nov%2012%202020&to=Nov%2019%202020/
:scheme: https
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
accept-encoding: gzip, deflate, br
accept-language: en-US,en;q=0.9
cookie: tdid=azJ6MW45bSUyQlRxMXhnMWlRdFlJbDJRJTNEJTNEOklyMk14M2tYVVpvSG9HUTRtJTJGZlFKdyUzRCUzRA; _ga=GA1.2.1783666257.1605728974; JSESSIONID=9C50C09FEE87F3CCF7701FC7C3F0F326; AWSALB=v4wU9BVN7zdWf0YrbhfTrsTRGXyV0x5VtFVhxHDMco7vIWs8SfIDrU9db00EbaakDwmEdE2pXltZSswTiEF/K069JdH6vMr4RvNYYpsSbsPUTVuUt/5NkLHTJEJd; AWSALBCORS=v4wU9BVN7zdWf0YrbhfTrsTRGXyV0x5VtFVhxHDMco7vIWs8SfIDrU9db00EbaakDwmEdE2pXltZSswTiEF/K069JdH6vMr4RvNYYpsSbsPUTVuUt/5NkLHTJEJd
sec-fetch-dest: document
sec-fetch-mode: navigate
sec-fetch-site: none
sec-fetch-user: ?1
upgrade-insecure-requests: 1
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36

Response Headers

cache-control: no-cache, no-store, max-age=0, must-revalidate
content-disposition: attachment; filename=client_Technical_Publications_stats.zip
content-security-policy: default-src 'none'; script-src 'self' 'unsafe-inline' 'unsafe-eval' data: blob:; connect-src 'self'; frame-src 'self'; frame-ancestors 'self'; img-src 'self' data: blob:; style-src 'self' 'unsafe-inline'; font-src 'self';
content-type: application/zip
date: Wed, 23 Dec 2020 17:22:55 GMT
expires: 0
feature-policy: geolocation 'self';midi 'none';sync-xhr 'self';microphone 'none';camera 'none';magnetometer 'none';gyroscope 'none';speaker 'self';fullscreen 'self';payment 'none';
pragma: no-cache
referrer-policy: no-referrer-when-downgrade
server: nginx/1.17.2
set-cookie: AWSALB=rYgsUIfM2f/cfdgdfgrf2SxsgFRrq58s0ChVFPOR7/zBzYdwb4/cRZYggtSXybifpD/J/0mBxH5kUIwVoDboy+KM8C3wN8o0HjUGCAjBg9qVIv2XA/r; Expires=Wed, 30 Dec 2020 17:22:55 GMT; Path=/
set-cookie: AWSALBCORS=rYgsUIfM2f/cQc9w1vbcvcvblQrf2SxsgFRrq58s0ChVFPOR7/zBzYdwb4/cRZYggtSXybifpD/J/0mBxH5kUIwVoDboy+KM8C3wN8o0HjUGCAjBg9qVIv2XA/r; Expires=Wed, 30 Dec 2020 17:22:55 GMT; Path=/; SameSite=None; Secure
status: 200
strict-transport-security: max-age=31536000; includeSubDomains
td-service: admin
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block

command

wget --header='Accept-Encoding: gzip, deflate,br' \
     --header='content-type: application/zip'\
     --header='Accept-Language: en-US,en;q=0.9'\
     --header='sec-fetch-mode: navigate'\
     --header='upgrade-insecure-requests: 1'\
     --header='sec-fetch-dest:document'\
     --header='sec-fetch-mode:navigate'\
     --header='scheme:https' \
     --user-agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36' \
     [email protected] \
     --password=secret! \
     -O ctp_stats.zip \
     "https://msi.domain.com/admin/ui/feedbackCSV/organizationReports/2Fsjf3j45345345bsdf?reports%5B%5D=pageviews&%20reports%5B%5D=searches&from=Nov%2012%202020&to=Nov%2019%202020/" 

I passed the same headers in my command , I'm getting 200 response code and below output Length: unspecified [text/html] Saving to: ‘ctp_stats.zip’ When I opened the zip folder I am seeing an empty file

1 Answer 1

0

Given that you're passing credentials to the web server that is generating this report, you will likely also need to pass some additional request header information such as:

wget --header='Content-Type: text/plain' \
     --header='Accept-Encoding: gzip, deflate' \
     --user-agent='User-Agent: Mozilla/5.0 (Windows; MSIE 5; Windows ME; en-US)' \
     [email protected] \
     --password=secret! \
     -O ctp_stats.zip \
     "https://msi.domain.com/admin/ui/feedbackCSV/organizationReports/sjf3j45345345bsdf?reports[]=pageviews&reports[]=searches&from=Nov 12 2020&to=Nov 19 2020/client_Technical_Publications_stats.zip"

You will probably want to use the Developer Tools from your favourite browser to see what HTTP headers are being sent when requesting this file so that you can fill in the appropriate data.

More information about all the different types of headers available via wget can be found on this page of the manual.

While it was not part of the question, there is also a -O option in there to output the file to ctp_stats.zip so that you don't get a crazy-long name that. Feel free to use/modify/delete this if it doesn't add any value.

1
  • Thanks, @Matigo,I find the headers using the chrome developer tool and I passed the same headers in my command, but it's writing an empty file to the zip folder
    – Rahul
    Commented Dec 23, 2020 at 18:00

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .