Copy File from HDFS to Local File System based on timestamp Using Shell Script

Question

I want to move n number of file residing in the Hadoop directory to Local File system based on timestamp. Scenario:

1. suppose I am doing the first time transition from HDFS to the Local file system, so there will be no file in Local file system also. in this case all file residing in HDFS directory will be moved to the local file system
2. Now if the new file comes in HDFS directory and I need to again move the file to the Local file system, in this case, latest file which not yet been processed those only will be transferred, based on timestamp.

I want to be created a automation script which does this task

preffered using shell script

Answer 1

As per the above scenario to do this task, there are various ways, we can do this on the basis of timestamp or by keeping track of processed file in some log file

Here I am using the approach in that I will keep track of the name of the file which has been processed i.e. moved from HDFS to the local file system.

if I am running script first time, the file which is residing in HDFS will be moved to the Local file system and check whether the name of the file is present in Processedfile.txt or not. If it is not then file will be moved to the Local file system and make the entry in Processedfile.txt. If filename is present in Processedfile.txt than won't move to the local file system.

Code:

# Local file system location
path=/home/cloudera/test/

#processedfile.txt location
logpath=/home/cloudera/test/processedfile.txt

#HDFS file location
hdfspath=/user/test/

# to get list of files from HDFS dir
for filename in `hadoop fs -ls ${hdfspath}*|  awk '{print $NF}'  | tr '\n' ' '`
do

# get file name
    name=$(basename $filename)

# check filename is present in processedfile.txt or not 
            if grep -Fxq $name $logpath;
        then
        echo $name is present in local file system
        else

#copy file from HDFS to Local File System.      
          hadoop fs -copyToLocal $filename $path
          echo "$name" >> "$logpath"
        fi
done
echo prosessing completed