1. The problem:
Series upload of 100 or more files via FTP is interrupted. Behavior irregular. On a new PC from 2011, Ubuntu just installed. No other problem with Ubuntu. Fast browser action and Website loading.
What could I try, in addition to the steps described here below?
Or is there a primitive error or a primitive solution?
Original text not modified. Updates done by adding:
- Dec.7, 2011: Information added below in 3.2)
- Dec.9,2011: added: 4.d) 4.e)
- Dec.10,2011: added 4.f)
2. More facts:
FTP upload of several 1000 files (several sites) is here done by own Perl programs which send a series of FTP standard commands to the standard ftp of the LINUX system.
Effect: Similar to mirroring - but enables fine-tuning.
The used single commands "ftp mput" comprise e.g. 100 files, each file approx. 100 KB. Approx. 1000 files per Website.
Uploading to: Websites using shared hosting. The servers use Linux, e.g. Bluehost=Hostmonster or Lunarpages.
3.1) So far I never had such a problem.
During years I never had such a problem with Fedora. Fedora 13 on a PC from 2006 continues to do the job even now properly (idential DSL connection, identical DSL hardware). I intentionally did not upgrade Fedora 13. So LINUX ftp might have changed a bit in the meantime. But it is not probable that this is the reason.
PCs are here only used as OS supplier. Data and own software are on an external HD, are OS-independent and portable (organized by Perl software).
The new UBUNTU PC does not do this job perfectly, while working on the identical external HD and with an identical software environment.
(Only the network cable differs, but this should not cause any problem. 1 m length for the old FEDORA PC, 5 m for the new Ubuntu PC.)
3.2) UBUNTU specifics cause probably the problem (added Dec.7,2011)
In the meantime / now I have :
UBUNTU 64b from late 2011 on the new PC / 2011, SATA hard disks.
UBUNTU 32b from late 2011 on the old PC / 2006, PATA (/IDE) hard disks.
These are just suppliers of the operating system. All data, software and execution is done on an external HD which is IDENTICAL for both cases.
In both cases, the described problem occurs. It does NOT occur for the old PC when run with Fedora 13 (from 2009 or so).
The problem is hence very probably somehow correlated to specific features of UBUNTU.
There is a very low probability that since FEDORA 13 the general LINUX system has changed, resulting in this problem.
With a rapid Google search, so far I did not find any information about similar problems.
Everything else of UBUNTU is working fine on the new PC with a wide screen.
So it is from now for a while here the choice.
I am getting these weeks a 10 times faster DSL Internet connection. Perhaps the problem will then disappear. (I suppose it will NOT disappear.)
4.a) Priority setting does not help.
With the command nice I have tested a priority of -18 (verified in gnome-system-monotor - yes, -18). This did not help.
4.b) sudo use did not help
I also tried to call the Perl program which does the job, with sudo. This did not influence the result.
4.c) Behaviour irregular
There is no rule at what point of time (which file) the upload is stopped. Normally after some minutes - only once it did the whole job of 2 hours. There is perhaps a minor correlation to the rush hours of Internet use. But this is not sure.
4.d) No help from: source code, -i flag, -v flag (added 9 Dec.2011)
The involved COMPLETE long Perl programs would not be helpful here. (Various features, sites, specifics,...).
Here the OWN command (Perl subroutine) with the upload problem (example):
&FTP_c_mput (" www/ppp-de/*.htm")
This simply does the FTP command : mput www/ppp-de/*.htm
for ~200 files, but (the problem:) halts at file 30 or so
-i All is automatic - hence already with -i flag, hence nothing interactive. So a timeout error should normally never occur.
-v --- Always in verbose mode (therefore the results in nr.5.2. and 5.3.)
I still have to implement within the software the debug function -d (like recommended). It is doubful that it will help - because...:
4.e) Most probable problem reason - I suppose (added Dec 9,2011)
Most probable is that some process, specific to Ubuntu, creates a delay - so that the ftp program does not supply fast enough the next file to the remote PC (Internet server). But I did not state delays of several seconds for any other function. Everything works perfect and fast. And I could not find any correlation of some software action or so before the problem occurrence.
I state per,anently a short hard disk access every 3 seconds (if the control led is acting properly). Every 3 seconds - or faster. Not regular. This may have simple explanations. This even continues when there is no application running - a totally blank monitor. But I do not believe that the reason is related to this.
4.f) Debug feature? ftp -d (added Dec.10,2011)
The -d flag for ftp : Not tried (while recommended by a comment).
The -d flag as such is not sufficient. To get ftp logged, various steps are required (rsyslog.conf...). The probability that this time investment would lead to success, is for this problem type close to zero.
So I will instead continue the work-around: Doing bulk uploading on Fedora (= PC 1), all the rest on Ubuntu (= PC2), and try other steps to find in some future the solution.
5.1. Example: This is a control display during uploads
This display is done by my own Perl programs, hence not by FTP. Shortened. May include 100...300 files.
www/xmed-ppp-de/index.htm www/xmed-ppp-de/wweb-pare-med-de.htm www/xmed-ppp-de/wwee-fina-med-de.htm www/xmed-ppp-de/wwfu-cont-med-de.htm www/xmed-ppp-de/wwfu-sepa-med-de.htm
5.2. Example: The standard display of the ftp program is like:
150 Connecting to port 63555
226-File successfully transferred
226 7.126 seconds (measured here), 10.14 Kbytes per second
73985 bytes sent in 0.81 secs (89.7 kB/s)
local: www/xmed-ppp-de/wyck-tob-bo-med-de.htm remote: www/xmed-ppp-de/wyck-tob-bo-med-de.htm
200 PORT command successful
5.3. Example: The display when interrupted:
With new installed UBUNTU on a new PC, the series is ending mostly after approx. 30 files, varying in random manner between 20 und 50, with the following error message: - hence: a Timeout error for MY typing - but it was the FTP bulk command mput , - so there can not really be a typing speed problem on MY PC...
150 Connecting to port 63562
226-File successfully transferred
226 10.317 seconds (measured here), 14.02 Kbytes per second
148068 bytes sent in 6.65 secs (21.8 kB/s)
local: www/xmed-ppp-de/wyck-tob-ris-med-de.htm remote: www/xmed-ppp-de/wyck-tob-ris-med-de.htm
200 PORT command successful
421 Timeout - try typing a little faster next time
local: www/xmed-ppp-de/wyck-tob-sto-med-de.htm remote: www/xmed-ppp-de/wyck-tob-sto-med-de.htm
No control connection for command: Success
local: www/xmed-ppp-de/wycu-nut-med-de.htm remote: www/xmed-ppp-de/wycu-nut-med-de.htm
............... etc. etc. for all following files ............