urllib.error.HTTPError: HTTP Error 404: Not Found in Docker from Ubuntu only

Question

I have a weird problem. Docker containers are supposed to provide their own environments, so if they work on one host platform then it should work on any other OS with Docker support as well, right? That's what I understood to be one of Docker's main selling points.

I am making use of an external pytorch based application through a provided Dockerfile. I extended it with REST endpoints to call its functions and am making a new image out of it. The Dockerfile looks like this:

FROM pytorch/pytorch

ARG gh_username=myUsername
ARG service_home="/home/EasyOCR"

# Configure apt and install packages
RUN apt-get update -y && \
    apt-get install -y \
    libglib2.0-0 \
    libsm6 \
    libxext6 \
    libxrender-dev \
    libgl1-mesa-dev \
    git \
    # cleanup
    && apt-get autoremove -y \
    && apt-get clean -y \
    && rm -rf /var/lib/apt/li

RUN mkdir "$service_home" \
    && git clone "https://github.com/$gh_username/EasyOCR.git" "$service_home" \
    && cd "$service_home" \
    && git remote add upstream "https://github.com/myUsername/EasyOCR.git" \
    && git pull upstream master

# Build
RUN cd "$service_home" \
    python setup.py build_ext --inplace -j 4 \
    && python -m pip install -e .

WORKDIR "$service_home/server"

COPY ./python-server/. "."
RUN pip install -r "requirements.txt"
EXPOSE 5050  # gunicorn is hosted on port 0.0.0.0:5050
ENTRYPOINT ["gunicorn", "--config", "gunicorn_config.py", "wsgi"]
CMD ["--gpus all"]

and docker-compose.yml:

easy-ocr:
  build: .
  image: myUsername/easy-ocr
  container_name: easy-ocr
  ports:
    - 5050:5050

Running this image on my Windows machine (with Docker Desktop WSL2) works fine. I can call the endpoints and get the expected responses. However, when I try to run this on Ubuntu 20.04 then I get the following runtime error in the container:

easy-ocr         | [2020-12-13 18:56:42 +0000] [8] [ERROR] Exception in worker process
easy-ocr         | Traceback (most recent call last):
easy-ocr         |   File "/opt/conda/lib/python3.8/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
easy-ocr         |     worker.init_process()
easy-ocr         |   File "/opt/conda/lib/python3.8/site-packages/gunicorn/workers/base.py", line 119, in init_process
easy-ocr         |     self.load_wsgi()
easy-ocr         |   File "/opt/conda/lib/python3.8/site-packages/gunicorn/workers/base.py", line 144, in load_wsgi
easy-ocr         |     self.wsgi = self.app.wsgi()
easy-ocr         |   File "/opt/conda/lib/python3.8/site-packages/gunicorn/app/base.py", line 67, in wsgi
easy-ocr         |     self.callable = self.load()
easy-ocr         |   File "/opt/conda/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 49, in load
easy-ocr         |     return self.load_wsgiapp()
easy-ocr         |   File "/opt/conda/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 39, in load_wsgiapp
easy-ocr         |     return util.import_app(self.app_uri)
easy-ocr         |   File "/opt/conda/lib/python3.8/site-packages/gunicorn/util.py", line 358, in import_app
easy-ocr         |     mod = importlib.import_module(module)
easy-ocr         |   File "/opt/conda/lib/python3.8/importlib/__init__.py", line 127, in import_module
easy-ocr         |     return _bootstrap._gcd_import(name[level:], package, level)
easy-ocr         |   File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
easy-ocr         |   File "<frozen importlib._bootstrap>", line 991, in _find_and_load
easy-ocr         |   File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
easy-ocr         |   File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
easy-ocr         |   File "<frozen importlib._bootstrap_external>", line 783, in exec_module
easy-ocr         |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
easy-ocr         |   File "/home/EasyOCR/server/wsgi.py", line 1, in <module>
easy-ocr         |     from app import app as application
easy-ocr         |   File "/home/EasyOCR/server/app.py", line 1, in <module>
easy-ocr         |     import ocr
easy-ocr         |   File "/home/EasyOCR/server/ocr.py", line 4, in <module>
easy-ocr         |     'ch_sim': easyocr.Reader(['ch_sim']),
easy-ocr         |   File "/home/EasyOCR/easyocr/easyocr.py", line 82, in __init__
easy-ocr         |     download_and_unzip(model_url['detector'][0], DETECTOR_FILENAME, self.model_storage_directory)
easy-ocr         |   File "/home/EasyOCR/easyocr/utils.py", line 540, in download_and_unzip
easy-ocr         |     urlretrieve(url, zip_path,reporthook=printProgressBar(prefix = 'Progress:', suffix = 'Complete', length = 50))
easy-ocr         |   File "/opt/conda/lib/python3.8/urllib/request.py", line 247, in urlretrieve
easy-ocr         |     with contextlib.closing(urlopen(url, data)) as fp:
easy-ocr         |   File "/opt/conda/lib/python3.8/urllib/request.py", line 222, in urlopen
easy-ocr         |     return opener.open(url, data, timeout)
easy-ocr         |   File "/opt/conda/lib/python3.8/urllib/request.py", line 531, in open
easy-ocr         |     response = meth(req, response)
easy-ocr         |   File "/opt/conda/lib/python3.8/urllib/request.py", line 640, in http_response
easy-ocr         |     response = self.parent.error(
easy-ocr         |   File "/opt/conda/lib/python3.8/urllib/request.py", line 569, in error
easy-ocr         |     return self._call_chain(*args)
easy-ocr         |   File "/opt/conda/lib/python3.8/urllib/request.py", line 502, in _call_chain
easy-ocr         |     result = func(*args)
easy-ocr         |   File "/opt/conda/lib/python3.8/urllib/request.py", line 649, in http_error_default
easy-ocr         |     raise HTTPError(req.full_url, code, msg, hdrs, fp)
easy-ocr         | urllib.error.HTTPError: HTTP Error 404: Not Found
easy-ocr         | [2020-12-13 18:56:42 +0000] [8] [INFO] Worker exiting (pid: 8)

It suddenly complains about the request.urllib package, but it had no problems when I ran this Docker image on Windows? I am sure that the Docker image is the same. The docker-compose version on Windows is 1.27.4 and on Ubuntu 1.25.0. I sincerely hope that's not a possible cause for this issue.

As a sanity check, I decided to run the python application outside of the Docker container and running gunicorn --config gunicorn_config.py wsgi and it works!

I am a total novice in Linux so I may be overlooking something basic. What could be the cause for this issue?

Answer 1

I found the answer and it's a very stupid one. The resource that is being retrieved doesn't exist... I forked a repository and the code references assets from the releases of the original repository. After some testing I changed the links in my forked repository to point to my own releases, which I thought included everything. All seemed well.

I then tried to run the same on Ubuntu for the first time and it fails, because the needed packages haven't been copied with the fork. Some have been copied and some haven't, so I am unsure what rules are in place for forks. Anyway, I was under the illusion I had them because Windows kept 'finding' them. So why did it keep succeeding on Windows despite me deleting the existing volumes and images? It apparently kept a cache of this line:

RUN mkdir "$service_home" \
    && git clone "https://github.com/$gh_username/EasyOCR.git" "$service_home" \
    && cd "$service_home" \
    && git remote add upstream "https://github.com/AlexanderTang/EasyOCR.git" \
    && git pull upstream master

Despite me changing the user for ARG gh_username to my own user, it still kept using the old value to pull from the repository. That means it was never going to my forked repository but to the original one.

Triggering a change on the line removes the cache and makes Docker re-evaluate the line. When I replace $gh_username explicitly with my own user (git clone "https://github.com/myUsername/EasyOCR.git"), it (correctly) gets the same error I am getting on Ubuntu.