What is aws s3?
Aws s3 stands for simple storage service provided by amazon web service.
Here are some of the use cases of aws s3
Static Website Hosting
you can host any static web site which is built in (react,nextjs,angular, vue, gatsby,remix).
Archive Datayou can archive much of the data with low cost.
Backups (most used)you can store your backups like db backup every day using cron job.
Also there are some additional advantage of it which you can explore in aws console
Let's understand how s3 works.
In order to use s3, you need to create a bucket where you want to store anything. you can compare it to folder where all files are stored.
Each bucket has unique name and you can create folder/files in the bucket. It stores data as key value pair. We will see in further that how it stores data in key value pair. S3 provides storage limit 0 to 5 TB.
Generally Root user of aws creates Identity and Access Management (IAM) user to provide access of aws with some rules.
IAM user can create bucket, and also can perform some operations on it.
In order to create IAM user, you need to create account on aws
Once you have created account, you can find IAM user creation option. After creating IAM user, create the s3 bucket using s3 option. See the below image.
Once you click on s3 you will find following screen.
Click on the create bucket and fill all the data, You will also need to select rules like permissions and all.
After creating a bucket aws will provide you Access key id and Secret access key.
Save those key in your machine because we will need those keys in order to perform any actions on it.
Now create a new nodejs project using following command. (I assume that you have downloaded node in your machine).
npm init -y
-y is a flag which is used when you want to give yes to all questions which is asked when you create node project.
Install few dependencies to create an app.
npm i express dotenv express-fileupload
Above dependecies are used to create express server and upload files. dotenv file is for storing the private keys provided by aws.
Now create index.js file with following code.
const express = require("express");
const app = express();
const fileUpload = require("express-fileupload");
//express middlewares
app.use(fileUpload({
useTempFiles:true,
tempFileDir:"/tmp"
}));
const PORT = process.env.PORT || 4000;
app.listen(PORT,()=>console.log(`server is running on port ${PORT}`));
We will need express file upload middleware to upload files, we also can create own functionality to create temporary folder and store file in folder, once it is uploaded we can delete that folder.
Let's create two folder, Rotues and utils.
In routes folder we will create routes of api and we will send response based on actions, We surely can create controller folder but as of our job can be done using route.
In utils (utility) folder we will create one file which is called s3.js to perform some actions based on s3.
GO to routes folder and create file with .js extension, Here I have created uploadFile.js
Write following thing in that file.
const router = require('express').Router();
const s3 = require("../utils/s3");
router.post("/upload",(req,res)=>{
const file = req.files.file;
s3.uploadToS3(file,(error,data)=>{
if(error){
return res.send({error:"Something went wrong."});
}
return res.send({message:"File uploaded successfully"});
});
});
module.exports = router;
Here we are having post method, We will get one file from front-end or you can use post man here.
s3.uploadToS3 is the function which we will create in s3.js file to upload file, we will also return call back with data and error.
Important
We will need to install aws sdk dependency to perform action in aws s3
So write following command
npm i aws-sdk
After Installing it , go to s3.js file and import following things.
const path = require('path');
const fs = require('fs');
const fileUpload = require('express-fileupload');
const aws = require('aws-sdk');
Now we will need to configure the aws to create connection between our app and aws.
//configure the aws environment
aws.config.update({
accessKeyId:process.env.AWS_ACCESS_KEY_ID,
secretAccessKey:process.env.AWS_SECRET_ACCESS_KEY,
});
Above code is just used to create environment of s3.
now create instance of s3 so we can use all methods of s3.
//initialize s3
const s3 = new aws.S3();
Now we can create our function to upload files which we are calling in our uploadFiles.js
exports.uploadToS3 = (file,next) =>{
const fileStream = fs.createReadStream(file.tempFilePath);
console.log(file.tempFilePath)
const params = {
Bucket:process.env.AWS_BUCKET_NAME,
Body:fileStream,
Key:file.name
};
s3.upload(params,(error,data)=>{
console.log(error,data);
next(error,data);
});
};
So we will need some parameters to send to s3, so s3 can know that where to upload file and in which bucket to.
We will send bucket name, Body means the data which we want to upload and key which you can give any name. I will just give the file name as key name.
s3.upload method accept parameters and based on it it return call back function of data and error.
We will send data and error in our callback function, which we are getting in uploadFiles.js
Now you can go to index.js and import following things.
const uploadFiles = require("./routes/uploadFiles");
//you use this as a moddleware of routes
//our base api route will be /api/v1/*
app.use("/api/v1",uploadFiles);
So final index.js will be look like this.
const express = require("express");
const app = express();
const fileUpload = require("express-fileupload");
//express middlewares
app.use(express.json());
app.use(express.urlencoded({extended:true}));
app.use(fileUpload({
useTempFiles:true,
tempFileDir:"/tmp"
}));
//all routes
const uploadFiles = require("./routes/uploadFiles");
//route middlewares
app.use("/api/v1",uploadFiles);
//config http server
const PORT = process.env.PORT || 4000;
app.listen(PORT,()=>console.log(`server is running on port ${PORT}`));
Now we can use our api in postman. localhost:4000/api/v1/upload is the url to can api.
See the image below to select configuration to send file.
I have selected image file so it will be uploaded on s3. You Also can verify on s3 whether it is uploaded or not.
Once it is uploaded we will get following metadata by s3.
ETag: 'some random string',
Location: your url of file,
key: 'horizontal_tagline_on_white_by_logaster.jpeg',
Key: 'horizontal_tagline_on_white_by_logaster.jpeg',
Bucket: your bucket name
We can store key and location in our db so in future we can select or delete anything based on key.
Now we can create download and delete file functionality which is much easier.
Since we need bucket name in all the parameter which we will pass in all methods, I will make and object so we don't need to write it again.
const constantParams = {
Bucket:process.env.AWS_BUCKET_NAME
}
So now Our s3.js file will look like this.
const path = require('path');
const fs = require('fs');
const fileUpload = require('express-fileupload');
const aws = require('aws-sdk');
//configure the aws environment
aws.config.update({
accessKeyId:process.env.AWS_ACCESS_KEY_ID,
secretAccessKey:process.env.AWS_SECRET_ACCESS_KEY,
});
//initialize s3
const s3 = new aws.S3();
//constant params
const constantParams = {
Bucket:process.env.AWS_BUCKET_NAME
}
//upload file to s3 bucker
exports.uploadToS3 = (file,next) =>{
const fileStream = fs.createReadStream(file.tempFilePath);
const params = {
...constantParams,
Body:fileStream,
Key:file.name
};
s3.upload(params,(error,data)=>{
console.log(error,data);
next(error,data);
});
};
//download file from s3 bucket
exports.getFileFromS3 = key =>{
const downloadParams = {
Key:key,
...constantParams
};
return s3.getObject(downloadParams).createReadStream();
};
//delete file from s3 bucker
exports.deleteFileFromS3 = (key,next) =>{
const deleteParams = {
Key:key,
...constantParams
};
s3.deleteObject(deleteParams,(error,data)=>{
next(error,data);
});
};
s3.getObject method used when we want to get any file from s3
s3.deleteObject method will delete the file.
As you can see in above code our function for download and delete is expecting the key, which we get by s3 so we can pass that key
Now our uploadFiles.js file will look like this.
const router = require('express').Router();
const s3 = require("../utils/s3");
router.post("/upload",(req,res)=>{
const file = req.files.file;
s3.uploadToS3(file,(error,data)=>{
console.log("commit")
if(error){
return res.send({error:"Something went wrong."});
}
return res.send({message:"File uploaded successfully"});
});
});
router.get("/getFile",async (req,res)=>{
try {
let fileToSend = await s3.getFileFromS3('horizontal_tagline_on_white_by_logaster.jpeg');
fileToSend.pipe(res);
} catch (error) {
res.send({error:"Server Error"});
}
});
router.delete("/deleteFile",(req,res)=>{
s3.deleteFileFromS3('horizontal_tagline_on_white_by_logaster.jpeg',(error,data)=>{
if(error){
return res.send({error:"Can not delete file, Please try again later"});
}
return res.send({message:"File has been deleted successfully"});
});
});
module.exports = router;
Now you can call both download and delete api, the end point of both api will be,
download file : localhost:4000/api/v1/getFile/:key
delete file : localhost:4000/api/v1/getFile/:key
I am passing the key directly because we didn't saved it in our db. but in real scenario we will be saving key in db and fetching keys to perform actions.
So this is how you can perform some actions on s3. Trust me it is very easy than it seems and also depends how you want to craft you app.
Hope you get something from this article, If you still confused at any point then please comment below.
Thanks :)
Top comments (0)