DEV Community

Cover image for Mastering Hadoop FS Shell: copyFromLocal and get Commands
LabEx profile image Labby
Labby for LabEx

Posted on

Mastering Hadoop FS Shell: copyFromLocal and get Commands

#labex #hadoop #coding #programming

Introduction

MindMap

In a galaxy far far away, there exists a ongoing space war between different factions. The galaxy is also bustling with space traders trying to make profits amidst the chaos. Our story focuses on one such space trader who needs to transfer valuable data files using Hadoop's HDFS file system. The trader's goal is to successfully copy files from their local system to Hadoop using the copyFromLocal command and retrieve files from Hadoop to their local system using the get command.

Copy File to Hadoop

In this step, the space trader needs to copy a file named data.txt from their local system to Hadoop HDFS.

Open the terminal and follow the steps below to get started.

  1. Switch to the hadoop user for proper permissions: 
   su - hadoop
Enter fullscreen mode Exit fullscreen mode
  1. Create a directory named space_data in Hadoop: 
   hdfs dfs -mkdir /space_data
Enter fullscreen mode Exit fullscreen mode
  1. Copy the local file data.txt to Hadoop: 
   hdfs dfs -copyFromLocal /home/hadoop/data.txt /space_data/
Enter fullscreen mode Exit fullscreen mode
  1. Verify that the file was successfully copied: 
   hdfs dfs -ls /space_data
Enter fullscreen mode Exit fullscreen mode

Retrieve File from Hadoop

Now, the space trader needs to retrieve the file data.txt from Hadoop and save it to their local system.

  1. Change to the directory where you want to save the retrieved file: 
   mkdir /home/hadoop/space_data && cd /home/hadoop/space_data
Enter fullscreen mode Exit fullscreen mode
  1. Retrieve the file data.txt from Hadoop to the local system: 
   hdfs dfs -get /space_data/data.txt
Enter fullscreen mode Exit fullscreen mode
  1. Verify that the file was successfully retrieved: 
   ls -l data.txt
Enter fullscreen mode Exit fullscreen mode

Summary

In this lab, we simulated a space trading scenario where a trader needed to transfer data files between their local system and Hadoop HDFS. By practicing the copyFromLocal and get commands in Hadoop FS Shell, users can learn how to efficiently move files to and from Hadoop's distributed file system. This hands-on experience helps in understanding the basic file management operations in Hadoop and enhances one's skills in working with big data ecosystems.

🚀 Practice Now: Hadoop FS Shell copyFromLocal/get

Want to Learn More?

Top comments (0)

Read next

msnmongare profile image

How to SSH into a Server Using IP Address, Username, and Password

Sospeter Mong'are -

szabgab profile image

Rust Source Code Reading: The thousands crate

Gabor Szabo -

vladislav_v_75b0533bb1e3 profile image

WebAssembly + JavaScript: Building a Real-Time Image Processing Tool

Vladislav V. -

ianbrumby profile image

Finally got some time to play with the new JSONata and Variables support for Step Functions, and I have to say, it is massive improvement. Check out my latest blog post, where I walk through a simple example of how easy it is to handle pagination now

Ian -