DEV Community

Cover image for Mastering Hadoop FS Shell: copyFromLocal and get Commands
LabEx profile image Labby
Labby for LabEx

Posted on

Mastering Hadoop FS Shell: copyFromLocal and get Commands

#labex #hadoop #coding #programming

Introduction

MindMap

In a galaxy far far away, there exists a ongoing space war between different factions. The galaxy is also bustling with space traders trying to make profits amidst the chaos. Our story focuses on one such space trader who needs to transfer valuable data files using Hadoop's HDFS file system. The trader's goal is to successfully copy files from their local system to Hadoop using the copyFromLocal command and retrieve files from Hadoop to their local system using the get command.

Copy File to Hadoop

In this step, the space trader needs to copy a file named data.txt from their local system to Hadoop HDFS.

Open the terminal and follow the steps below to get started.

  1. Switch to the hadoop user for proper permissions: 
   su - hadoop
Enter fullscreen mode Exit fullscreen mode
  1. Create a directory named space_data in Hadoop: 
   hdfs dfs -mkdir /space_data
Enter fullscreen mode Exit fullscreen mode
  1. Copy the local file data.txt to Hadoop: 
   hdfs dfs -copyFromLocal /home/hadoop/data.txt /space_data/
Enter fullscreen mode Exit fullscreen mode
  1. Verify that the file was successfully copied: 
   hdfs dfs -ls /space_data
Enter fullscreen mode Exit fullscreen mode

Retrieve File from Hadoop

Now, the space trader needs to retrieve the file data.txt from Hadoop and save it to their local system.

  1. Change to the directory where you want to save the retrieved file: 
   mkdir /home/hadoop/space_data && cd /home/hadoop/space_data
Enter fullscreen mode Exit fullscreen mode
  1. Retrieve the file data.txt from Hadoop to the local system: 
   hdfs dfs -get /space_data/data.txt
Enter fullscreen mode Exit fullscreen mode
  1. Verify that the file was successfully retrieved: 
   ls -l data.txt
Enter fullscreen mode Exit fullscreen mode

Summary

In this lab, we simulated a space trading scenario where a trader needed to transfer data files between their local system and Hadoop HDFS. By practicing the copyFromLocal and get commands in Hadoop FS Shell, users can learn how to efficiently move files to and from Hadoop's distributed file system. This hands-on experience helps in understanding the basic file management operations in Hadoop and enhances one's skills in working with big data ecosystems.

🚀 Practice Now: Hadoop FS Shell copyFromLocal/get

Want to Learn More?

Top comments (0)

Read next

atharva3000 profile image

OpenBB vs Proprietary Tools: Why Open Source is the Future of Financial Analysis

Atharva_404 -

niuweme profile image

This is why i hate project based learning

Manu Nunez -

dev_frank profile image

DATATYPES IN C (double, long double, void, bool)

Frank -

mayank_mohapatra profile image

Creating a Multilingual Expense Tracker with Tolgee

Mayank Mohapatra -