Appwrite Storage meets limitless S3

Appwrite is an open source backend-as-a-service that abstracts all the complexity involved in building a modern application by providing you with a set of REST APIs for your core backend needs. Appwrite handles user authentication and authorization, realtime databases, cloud functions, webhooks, and much more! If anything is missing, you can easily extend Appwrite using your favorite backend language.

One of the core functionalities of Appwrite is Appwrite Storage. It allows you to upload, view, download, and query your project files. Appwrite Storage not only takes care of encryption, compression and antivirus scans, it’s also built on top of Appwrite’s flexible, yet simple permission system. Appwrite lets you store any files such as text documents, icons, images, videos and more.

🚀 What is new in Appwrite 0.13

The Appwrite 0.13 release offers a much more powerful Storage service! You can now group your files into Storage Buckets for better organization, as well as better control over allowed size and extension. On each bucket, you can also specify a different set of permissions, and toggle additional features such as encryption and compression.

Additionally, you can now upload files with no size limitation. In this release, chunked upload is supported and you can split your file into 5MB chunks to gradually upload a file as large as your bucket settings allow. Don’t worry, chunk upload logic is part of our SDKs and if it sees a file larger than 5MB, chunk upload will automatically kick in. One more secret! Our SDKs now accept an onProgress callback that will be triggered each time a new chunk is successfully processed. Implementing progress bars have never been easier 💪

Finally, Appwrite Storage is no longer limited by the size of your hard drive! With Appwrite 0.13, you can now connect the Storage service with cloud storage providers such as Amazon S3 or DigitalOcean Spaces. With the introduction of these adapters, you no longer need to worry about running out of space, or hitting bandwidth limits due to Appwrite Storage. More providers are coming soon, so stay tuned 😎In this guide, we’ll take a look at setting up Appwrite Storage using AWS S3 as the storage adapter.

🌊 Amazon S3 setup

Before we jump into Appwrite, let’s prepare your Amazon S3 bucket. After logging into the AWS Console:

1. Use the search bar to find the S3 service with a bucket icon on a green background and click the orange button saying Create bucket, you will be redirected to the Create bucket page.
2. Next, configure the Appwrite file storage. Most of the options are up to you, but make sure you keep the Block all public access toggle enabled - you do want to block all public access. _(NOTE: I would not recommend enabling server-side encryption, as this feature is supported by Appwrite and you can control it from there.) _
3. Finally, make sure to name your bucket with a unique name. Quick tip, this name will only be stored in the .env file of your Appwrite instance, and users will never see it. That means, the name can be as simple or as complex as you want.

Make sure to note down the Bucket name and AWS Region, we will need them later when connecting Appwrite to Amazon S3. In my case, they were appwrite-project-x-bucket and eu-central-1, as seen in the screenshot above.

Last but not least, you need to get a set of access keys to allow Appwrite to read and write from our newly created bucket. To do that, click on the username in the upper right corner, and then select Security credentials. Open the Access keys section and click on a blue button Create New Access Key. In the modal, click on Show Access Key, and you will be presented with two keys. The first one is called Access Key, and the one below is Secret Key. Please note both of them down, but be aware, the secret key is a really sensitive piece of information and AWS won’t show it to you in future.

That’s it! We have successfully set up our Amazon S3, and we have all the required credentials in our hands. Let’s look at how easy it is to connect Appwrite to your newly created bucket.

🧰 Connecting Appwrite to Amazon S3

Before you start, make sure you have the Appwrite instance up and running. Installation of Appwrite is as simple as running one command:

docker run -it --rm \
    --volume /var/run/docker.sock:/var/run/docker.sock \
    --volume "$(pwd)"/appwrite:/usr/src/code/appwrite:rw \
    --entrypoint="install" \
    appwrite/appwrite:latest

To learn more about the installation process, you can check out our installation guide.

Once our Appwrite instance is up and running, we can configure the storage adapter. To do that, let’s enter the .env file and locate _APP_STORAGE_DEVICE. Currently, it’s set to Local, but since we want to use Amazon S3, we change it to S3. With the new storage provider, we are now required to add new variables:

_APP_STORAGE_DEVICE=S3
_APP_STORAGE_S3_BUCKET=YOUT_BUCKET
_APP_STORAGE_S3_REGION=YOUR_REGION
_APP_STORAGE_S3_SECRET=YOUR_ACCESS_SECRET
_APP_STORAGE_S3_ACCESS_KEY=YOUR_ACCESS_KEY

Where do we get this information from? 😨 First of all, you can already fill in access and secret keys, these are the ones we got from earlier steps when creating security credentials in Amazon AWS Console. You should also have a bucket name and region from the bucket creation process.

Once you have all of these environment variables configured, we can save the file and restart appwrite using docker-compose up -d. Did you notice that Docker is smart enough to only restart containers that use variables you just changed? 🤯

Once the Appwrite instance is restarted, create a new account and a new project. In the left menu, select ‘Storage’ and create a new bucket. Finally, upload a file into your bucket.

You can see the file was successfully uploaded, and can start using Appwrite just like usual! To confirm you are actually talking to Amazon S3, visit your storage and you'll see the file in there.

Congrats! You have successfully connected Appwrite to Amazon S3 🥳 If you plan to use files directly from your amazon bucket, there are a few things to keep in mind:

1. Every file is compressed using GZIP compression before storing it in Amazon S3. You will need to decompress the file after download if you plan to use it.
2. By default, encryption is enabled on newly created buckets in Appwrite Storage. This means you won’t be able to view the file without decrypting it first. I recommend you disable encryption if you plan on downloading the files from your Amazon S3.
3. All compression, encryption and antivirus are skipped for files above 20MB. These files are stored in your Amazon S3 as plain files.

👨‍🎓 Conclusion

The worst nightmare for every project is bad scalability of their application. Thanks to the newly released provider system for Appwrite Storage service, you can now connect Appwrite with external storage providers instead of storing the files on your system. This prevents hitting a hard drive and bandwidth limits, as well as lets you use your favorite provider alongside Appwrite. And as you have seen in the tutorial above, you can easily connect Appwrite with Amazon S3 in just a few steps!

If you have a project to share, need help or simply want to become a part of the Appwrite community, I would love for you to join our official Appwrite Discord server. I can’t wait to see what you build!

📚 Learn more

You can use the following resources to learn more and get help:

• 🚀 Appwrite Github
• 📜 Appwrite Docs
• 💬 Discord Community