0

I have a dual boot with Linux 16.04 and Windows 10 on an Inspiron 15 3000 series and it seems that I ran out of memory while calling for a NumPy object. I tried to download DrQA, a system for reading comprehension applied to open-domain question answering developed by Facebook research group. It included to call for the provided trained models and data for Wikipedia question answering. I tried to run the interactive.py to try it but after asking my first question to the prompt but it's where it ran short of memory.

(wiki_env) mike@mike-thinks:~/Programing/WikiQueri/DrQA$ python3.6 scripts/pipeline/interactive.py 
07/19/2018 01:49:54 PM: [ Running on CPU only. ]
07/19/2018 01:49:54 PM: [ Initializing pipeline... ]
07/19/2018 01:49:54 PM: [ Initializing document ranker... ]
07/19/2018 01:49:54 PM: [ Loading /home/mike/Programing/WikiQueri/DrQA/data/wikipedia/docs-tfidf-ngram=2-hash=16777216-tokenizer=simple.npz ]
Traceback (most recent call last):
  File "scripts/pipeline/interactive.py", line 69, in <module>
    reader_model=args.reader_model,
  File "/home/mike/Programing/WikiQueri/DrQA/drqa/pipeline/drqa.py", line 109, in __init__
    self.ranker = ranker_class(**ranker_opts)
  File "/home/mike/Programing/WikiQueri/DrQA/drqa/retriever/tfidf_doc_ranker.py", line 37, in __init__
    matrix, metadata = utils.load_sparse_csr(tfidf_path)
  File "/home/mike/Programing/WikiQueri/DrQA/drqa/retriever/utils.py", line 34, in load_sparse_csr
    matrix = sp.csr_matrix((loader['data'], loader['indices'],
  File "/home/mike/.local/lib/python3.6/site-packages/numpy/lib/npyio.py", line 235, in __getitem__
    pickle_kwargs=self.pickle_kwargs)
  File "/home/mike/.local/lib/python3.6/site-packages/numpy/lib/format.py", line 674, in read_array
    array = numpy.ndarray(count, dtype=dtype)
MemoryError
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 63, in apport_excepthook
    from apport.fileutils import likely_packaged, get_recent_crashes
  File "/usr/lib/python3/dist-packages/apport/__init__.py", line 5, in <module>
    from apport.report import Report
  File "/usr/lib/python3/dist-packages/apport/report.py", line 30, in <module>
    import apport.fileutils
  File "/usr/lib/python3/dist-packages/apport/fileutils.py", line 23, in <module>
    from apport.packaging_impl import impl as packaging
  File "/usr/lib/python3/dist-packages/apport/packaging_impl.py", line 23, in <module>
    import apt
  File "/usr/lib/python3/dist-packages/apt/__init__.py", line 23, in <module>
    import apt_pkg
ModuleNotFoundError: No module named 'apt_pkg'

Original exception was:
Traceback (most recent call last):
  File "scripts/pipeline/interactive.py", line 69, in <module>
    reader_model=args.reader_model,
  File "/home/mike/Programing/WikiQueri/DrQA/drqa/pipeline/drqa.py", line 109, in __init__
    self.ranker = ranker_class(**ranker_opts)
  File "/home/mike/Programing/WikiQueri/DrQA/drqa/retriever/tfidf_doc_ranker.py", line 37, in __init__
    matrix, metadata = utils.load_sparse_csr(tfidf_path)
  File "/home/mike/Programing/WikiQueri/DrQA/drqa/retriever/utils.py", line 34, in load_sparse_csr
    matrix = sp.csr_matrix((loader['data'], loader['indices'],
  File "/home/mike/.local/lib/python3.6/site-packages/numpy/lib/npyio.py", line 235, in __getitem__
    pickle_kwargs=self.pickle_kwargs)
  File "/home/mike/.local/lib/python3.6/site-packages/numpy/lib/format.py", line 674, in read_array
    array = numpy.ndarray(count, dtype=dtype)
MemoryError

So I learnt that mean I run out of RAM, what could I do to cope with this error ? Is it because of the model being too big ?

Here is my memory usage from the output of free -m:

              total        used        free      shared  buff/cache   available
Mem:           7888        5515        1013         320        1359        1718
Swap:             0           0           0

and

mike@mike-thinks:~/Programming/WikiQueri/DrQA$ cat /proc/meminfo
MemTotal:        8078272 kB
MemFree:          291604 kB
MemAvailable:    2326220 kB
Buffers:          262392 kB
Cached:          2212332 kB
SwapCached:            0 kB
Active:          3691168 kB
Inactive:        1485432 kB
Active(anon):    2727772 kB
Inactive(anon):   274408 kB
Active(file):     963396 kB
Inactive(file):  1211024 kB
Unevictable:     2238236 kB
Mlocked:         2238252 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:               104 kB
Writeback:             0 kB
AnonPages:       4940072 kB
Mapped:           508460 kB
Shmem:            275728 kB
Slab:             243928 kB
SReclaimable:     158948 kB
SUnreclaim:        84980 kB
KernelStack:       13024 kB
PageTables:        49456 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     4039136 kB
Committed_AS:   11909484 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
HardwareCorrupted:     0 kB
AnonHugePages:      2048 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      161648 kB
DirectMap2M:     7088128 kB
DirectMap1G:     2097152 kB

I need about 16 GB of RAM required I don't know how to read these table. Do you know if I can make it ?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.