0

I would like to split a large file (300GB), which is in my home directory under the dev/sdb1 into two (or more) different parts and move them into two different disk drives. However, I would like my system to see the file as one.

To be more concrete I would like to split the file in two 150GB parts (or three 100GB, etc) and move those parts into different SSDs that I have. But, this file is used as an imput to a research project and I would like the system to see it as one. Additionally, I will need to specify its path in another file (a .prototxt file where I specify all the inputs to my program) and that is the reason why I want to have one path under which the system recognizes that (splitted) file. Unfortunately, the SSDs are not huge enough to hold the whole file.

Additionaly, when I run df -h the SSDs are show as: 

Filesystem                           Mounted on
cluster-name-1-int:/var/tmp/local    /import/cluster-name-1-int  
cluster-name-2-int:/var/tmp/local    /import/cluster-name-2-int

My system is ina cluster but I do not think that this will make any difference because I have access from the cluster-name-1 machine to the SSD on the others.Lastly, I do not have root access, so please tell me if your solution needs root privilages so that I would know and find a way to implement that solution.

EDIT: I provide as many information as I think would be useful. Use whatever you think might help towards a solution and any advice is welcome.

Improve this question
4
  • Can you format the SSDs? If so, make a RAID0 of them. How do you specify the path? If it is a bash shell script, we could use process substitution.
    – muru
    Aug 13, 2015 at 16:13
  • It's a shared system and I do not think that I can do anything because I do not have root privilages. How do I know if it is RAID already? the disks seems separate when I execute df -h?
    – Grey
    Aug 13, 2015 at 16:16
  • If you can access the disks separately, they're not in a RAID. So, the other option. How do you specify the path? Is it a shell script?
    – muru
    Aug 13, 2015 at 16:21
  • No it is not, it is defined in a .prototxt file. When the program runs, it uses information from the .prototxt file to get the information that it needs. One of those inputs is the path where this file is saved.
    – Grey
    Aug 13, 2015 at 16:34

1 Answer 1

Reset to default
2

This might be a completely bogus answer, or it might turn out to be absolutely brilliant. I haven't decided yet.

Create 3 101GB (100 + 1 for some extra wiggle room) disk image files, and place the disk image files on the respective drives. Then put those disk images in RAID. You should be able to treat "/media/[drive]/image1.img" just like /dev/sda1 in a RAID configuration. That is, create these image files:

/media/SSD1/image1.img
/media/SSD2/image2.img
/media/SSD3/image3.img

Or what ever image format you need to use. Like ".iso" or ".dmg" or whatever.

Then create a RAID array using /media/SSD1/image1.img instead of /dev/sda1, /media/SSD2/image2.img instead of /dev/sdb1, etc.

You should be able to do this with as many image files as you need. Just make sure to add a little extra space for room after formatting.

Improve this answer
7
  • I think it's brilliant! I also think that the OP asked the wrong question and should do hardware RAIDing, but this is the perfect answer to the question! ;-)
    – Fabby
    Aug 17, 2015 at 8:28
  • He did say he can't do HW RAID sine it's the University's machine.
    – Daniel
    Aug 17, 2015 at 16:31
  • Ah! Missed that! ;-)
    – Fabby
    Aug 17, 2015 at 17:11
  • But do you think that my idea will work?
    – Daniel
    Aug 17, 2015 at 21:30
  • Because I honestly have no idea
    – Daniel
    Aug 17, 2015 at 21:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.