1

I tried executing this code in windows and it ran flawlessly but in Ubuntu as soon as I run this code, Ubuntu freezes for 3-4 mins and then the result come and then Ubuntu behaves laggy until I restart it.

import numpy as np
import pandas as pd
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn import preprocessing
from xgboost import XGBClassifier
import xgboost as xgb
from sklearn.metrics import accuracy_score

dataset_len = 40000000
dlen = int(dataset_len/2)
X_11 = pd.Series(np.random.normal(2,2,dlen))
X_12 = pd.Series(np.random.normal(9,2,dlen))
X_1 = pd.concat([X_11, X_12]).reset_index(drop=True)
X_21 = pd.Series(np.random.normal(1,3,dlen))
X_22 = pd.Series(np.random.normal(7,3,dlen))
X_2 = pd.concat([X_21, X_22]).reset_index(drop=True)
X_31 = pd.Series(np.random.normal(3,1,dlen))
X_32 = pd.Series(np.random.normal(3,4,dlen))
X_3 = pd.concat([X_31, X_32]).reset_index(drop=True)
X_41 = pd.Series(np.random.normal(1,1,dlen))
X_42 = pd.Series(np.random.normal(5,2,dlen))
X_4 = pd.concat([X_41, X_42]).reset_index(drop=True)
Y = pd.Series(np.repeat([0,1],dlen))
df = pd.concat([X_1, X_2, X_3, X_4, Y], axis=1)
df.columns = ['X1', 'X2', 'X3', 'X_4', 'Y']
df.head()


9
  • 1
    Please add to your question information on the ways in which you have already tried to solve your issue. A simple search gives multiple pages with possible solutions. Which ones have you already tried, and why didn't they work in your case?
    – zwets
    Jul 30, 2021 at 9:21
  • this can be memory problem where the amount of ram needed for the code is taking up too much space try to debug on that s i had executed this on my ubuntu and it ran pretty flawlessly so , can you give this a check ? Jul 30, 2021 at 10:19
  • 1
    Can you run free -m before and after executing the code? Edit your question to add this information.
    – vidarlo
    Jul 30, 2021 at 10:40
  • @vidarlo yeah i ran free -m and it is showing i have only 600 mb free out of 7801 mb ram. How can i have more free ram?
    – New
    Jul 30, 2021 at 19:15
  • Please edit your question with the output before and after, as requested. You should look at available, not free for the amount of RAM that's actually available.
    – vidarlo
    Jul 30, 2021 at 19:34

1 Answer 1

0

The results of free -m after running the code shows that you've filled your swap space. This is bad; your system is effectively out of memory at that point. It can't write any more data to swap, and it has to start killing processes to make memory available.

Windows has a dynamic swap size, where the pagefile expands as needed. Linux has a bit more static approach to swap, where it's fixed and pre-allocated. This probably leads to the behaviour you see, because Ubuntu starts killing of processes as you're out of memory.

You can increase the available swap space to make things a bit better, but ultimately you need more RAM to run memory-intensive calculations.

2
  • Oh damn, I just decreased data_len value and it was very fast in ubuntu, I will increase the swap size if i will need in future, I was just checking Ubuntu speed with that kind of value.
    – New
    Jul 31, 2021 at 7:44
  • Increasing swap won't make thing fast; it's dead slow compared to RAM, but it may stop your system from actually crashing.
    – vidarlo
    Jul 31, 2021 at 8:07

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .