Home Scripting Languages Node.js Building Scalable Cloud Storage like AWS S3 Bucket Service

Building Scalable Cloud Storage like AWS S3 Bucket Service

April 17, 2023

630

NodeJS - AWS S3 Bucket Like Service without using AWS

Now a days Cloud Storage has became an essential part of a modern world in software development, There are various third party service that provide cloud storage like AWS S3, Google Cloud Storage, Microsoft Azure Blob Storage, OpenStack Swift and more. But what if you want to build your own cloud storage system that can be crucial to make sure that the cloud storage service application runs smoothly and efficiently. In this Article we will learn how to build a scalable cloud storage system using NODEJS.

How to build cloud storage using NodeJS at backend

Building a fresh new cloud storage backend application using NodeJS is an challenging task, but no worry it’s possible to create our own NodeJS AWS S3 bucket like service without using any third party services.

Here are step to be follow:

System Requirement: Start planning a system configuration that is need to work the cloud storage service smoothly, you need to configure system or Cloud Instance like RAM, Storage(amount of storage), Server speed, bandwidth and type of data user can store.
Project Architecture: Based on the System Requirement, you can design the architecture on your cloud storage system project, Like you must decide on type of servers & where you are going to store the data(storage space), the mechanism to retrieval of data and a API for accessing system.
Database: Select a database so that you can store metadata of files and folders that will get uploaded by the user. In our project we are using MongoDB a NoSQL database to store the record of upload files.
Implement Features: Implement features such as creating a folder/bucket, getting list of all the folders, getting a list of files in a folder, downloading uploaded files from a folder using an API, and deleting a file from a given folder.
Test and deployment: Test the cloud storage system to ensure it works as expected and deploy it to a server or cloud platform server instance.

Note: Building a secured cloud storage system from scratch is a complex process and you may need a complete dedicated team to make it secured. This project is for learning purpose only.

Creating a Cloud Storage Service Similar to AWS S3 Bucket using NodeJS

How to create node js project

Create a folder & open it in visual studio code, then to make it a NodeJS project run below command:

node init -y

the above command will convert a normal folder into nodejs project where it will create a file by name name “package.json”.

Installing node libraries

Below are list to nodejs libraries/module that are required to build cloud storage system:

Express: Express is basically used in Node.js for creating application at backend and is used to created API’s.
fs(file system): The fs (file system) module used to interact with file system like accessing file, reading content of file and writing to a file, FS library is an in-build NodeJS library to interact with system files.
body-parser: The body-parser is middleware functionality used in Node.js applications usually used to parse incoming request bodies in a middleware before your handlers, and then allowing you to access request data in a more convenient way.
crypto-js: Crypto-js is a node.js library basically used for data encryption and hashing technique it support algorithms such as AES,MD5-SHA-1, SHA-256 and more.
path: The path module is used to get the file & directory paths in nodeJS.
mongoose: Mongoose is a NodeJS Mongodb library that helps in communication with MongoDB database, it’s helps in creating schema model and to interact with MongoDB database server. Used to perform CRUD operation with mongodb database.
multer: Multer is used to handle all the incoming file uploads, Multer is an middleware using which we can upload single and multiples files.

run below command to install all the above libraries into your nodejs project:

npm install express fs body-parser crypto-js path mongoose multer

NodeJS Project Structure for build cloud storage system

Here in my node.js project I have create folder to give a proper structure to node project:

model: Here we have 2 database schema model i.e. user model & upload model.

middleware: In middleware folder will have functionality like user authentication using APIKEY & multer middleware that will help you in uploading files.

config: In this folder will have database connectivity configuration.

bucketFolder: is a root folder where user can create folders and upload files

routes: In router we will make use of express to create API routers that can perform event like Create a Bucket/Folder, Getting List of Bucket/Folder, Get list of files from particular bucket, upload file, download files and delete files.

API Routers created using Express routers

Below are the api route that will help user in performing event like:

Creating a Bucket / Folder.
Getting list of all bucket / Folder.
Uploading Files to a Bucker / Folder.
Getting list of Files from a particular bucket / Folder.
Downloading Files from a Bucket.
Deleting a file from a bucketing
Deleting a bucket / Folder.

1. NodeJS API to Create a Folder

Video Tutorial

The below code will create a user desired folder. To achieve this we will make use of fs library with fs.mkdirSync(<folderName>) to create a folder. In below code first I am checking if folder exist with same name or no using fs.existsSync(), if not then use fs.mkdirSync then create.

const fs = require('fs');

// Create a Bucket 
router.post("/createFolderBucket", Auth.userAuthMiddleWare,async (req, res) => {
    const folderName = req.body.folderName;
    if (!folderName) {
        return res.json({ status: false, message: "Folder Name is Mandatory" });
    }
    const rootFolder = "bucketFolder";
    const folderpath = `${rootFolder}/${folderName}`;
    try {
        if (fs.existsSync(rootFolder)) {
            if (!fs.existsSync(folderpath)) {
                fs.mkdirSync(folderpath);
                return res.json({ status: true, success: "Directory created" });
            }
        } else {
            fs.mkdirSync(rootFolder);
            if (!fs.existsSync(folderpath)) {
                fs.mkdirSync(folderpath);
                return res.json({ status: true, success: "Directory/Folder created" });
            }
        }
        return res.json({ status: true, success: "Directory/Folder Already Exist" });
    } catch (error) {
        console.log(error);
    }
});

2. NodeJS API to get list of Folders

Video Tutorial

The below code will give us list of all folder, For this we will make use of fs.readdir() function to read files & folder exist within a given path and then we need to filter out and check if the file is directory or a file by using fs.statSync(filePath).isDirectory();.

const fs = require('fs');

// get Bucket List
router.get("/getAllFolderBucket", Auth.userAuthMiddleWare, async (req, res) => {
    //joining path of directory 
    const directoryPath = path.join('bucketFolder');
    
    //passsing directoryPath and callback function
    fs.readdir(directoryPath, (err, files) => {
        if (err) {
            console.error(err);
            return;
        }

        const directories = files.filter(file => {
            const filePath = path.join(directoryPath, file);
            return fs.statSync(filePath).isDirectory();
        });

        console.log(directories);
        return res.json({ status: true, success: directories });
    });
});

3. Node.JS API to Upload Any kind of file/document into a folder

Video Tutorial

The below api router has a ability to upload any kind of file into a folder, Here to upload a file we have made use of multer middleware that will upload a single file to a desired bucket/folder and the same uploaded file details(metadata) record is been stored into mongodb database.

const fs = require('fs');
const { upload } = require('../config/multerConfig')
const UploadModel = require('../model/uploads.model');

//Upload Files to a Bucket and store same to Mongo DB
router.post("/uploadFileInBucket", Auth.userAuthMiddleWare, upload().single("myFile"), async (req, res) => {
    if (req.file) {
        const filefullPath = req.file.destination + req.file.filename;
        const uploaded = new UploadModel({ userId: req.user._id, filename: req.file.filename, mimeType: req.file.mimetype, path: filefullPath });
        await uploaded.save();
        res.json({ status: true, success: "File Uploaded Successfully" });
    }
});

4. Getting list of all Files from a particular bucket / Folder.

Video Tutorial

The below code will give us list of all the files that exist in a folder, For this we will make use of fs.readdir() function to read files & folder exist within a given path and then we need to filter out the files by check if the file is a directory or a file by using fs.statSync(filePath).isFile().

// get list of all files from a Particular bucket
router.get("/getAllFilesFromParticularBucket", Auth.userAuthMiddleWare, async (req, res) => {
    try {
        const bucketName = req.body.bucketName;
        const directoryPath = path.join(`bucketFolder/${bucketName}`);
        fs.readdir(directoryPath, (err, files) => {
            if (err) {
                return res.json({ status: false, message: `${bucketName}, No Such Bucket Found` });
            }

            const allfiles = files.filter(file => {
                const filePath = path.join(directoryPath, file);
                return fs.statSync(filePath).isFile();
            });

            console.log(allfiles);
            return res.json({ status: true, filesList: allfiles });
        });
    } catch (error) {
        console.log("error------->>", error);
    }
});

5. Node.JS API to download a file from a bucket

Video Tutorial

The below API is but different, user can pass the folderName and fileName in API URL itself, and then the file will get downloaded from a directory by using fs.createReadStream with pipe response function.

//download a Files from a Bucket
router.get('/downloadFile/:filename/:folderName', Auth.userAuthMiddleWare, (req, res) => {

    const { filename, folderName } = req.params;
    const filePath = `bucketFolder/${folderName}/${filename}`;

    if (!fs.existsSync(filePath)) {
        return res.status(404).json({ message: 'File not found' });
    }
    res.setHeader('Content-Disposition', `attachment; filename="${filename}"`);
    const fileStream = fs.createReadStream(filePath);

    fileStream.pipe(res);
});

6. Node.JS API to delete a file from a folder

Video Tutorial

To delete any file from a folder in nodejs we make use of fs library unlink() function, that will simple delete a files.

// This will delete Files from a given Bucket/folder
router.post("/deleteFileBucket", async (req, res) => {
    const folderName = req.body.folderName;
    const fileName = req.body.fileName;
    if (!folderName) {
        return res.json({ status: false, message: "Folder Name is Mandatory" });
    }
    if (!fileName) {
        return res.json({ status: false, message: "File Name is Mandatory" });
    }
    const rootFolder = "bucketFolder";
    const folderpath = `${rootFolder}/${folderName}/${fileName}`;
    try {
        fs.unlink(folderpath, (err) => {
            if (err) {
                console.error(err);
                return;
            }
            console.log('File deleted successfully');
        });
        return res.json({ status: true, success: "File deleted successfully" });
    } catch (error) {
        return res.json({ status: false, success: error });
    }
});

7. Nodejs API to delete a folder

Video Tutorial

Note: the below api can only delete a folder is if it’s empty, if the folder is no empty then you need to first make it empty and then delete the folder.

To delete a folder we can make use of fs.rmdirSync() function, here you just need to pass folder path.

// Delete Folder/Bucket only if it Empty
router.post("/deleteFolderBucket", async (req, res) => {
    const folderName = req.body.folderName;
    if (!folderName) {
        return res.json({ status: false, message: "Folder Name is Mandatory" });
    }
    const rootFolder = "bucketFolder";
    const folderpath = `${rootFolder}/${folderName}`;
    try {
        if (fs.existsSync(rootFolder)) {
            if (fs.existsSync(folderpath)) {
                fs.rmdirSync(folderpath);
                return res.json({ status: true, success: "Directory Deleted" });
            }
        }
        return res.json({ status: false, success: "Directory Not Found" });
    } catch (error) {
        return res.json({ status: true, success: "Directory can't be deleted because it Not Empty" });
    }
});