This content has been automatically translated from Ukrainian.
Sooner or later, almost every project that stores user files - avatars and other media - faces the same question: do we really need to continue paying a cloud provider for every byte of traffic and every HTTP request?
This is especially relevant for small services. The application is already running on its own VPS, which has a significant amount of disk space, processing power, and bandwidth. Meanwhile, a separate service for files continues to charge for every GET request, every gigabyte of outgoing traffic, and additionally - a CDN in front of it.
In this article, I will conceptually explain how to migrate file storage from Amazon S3 to your own server without rewriting the application or changing the business logic. We will look at the architecture, security model, real issues during migration, and a checklist to go through before finally shutting down the old bucket.
Why at the conceptual level? There are many tools, and the approaches are universal. I recently migrated files from Amazon S3 to SeaweedFS (a container on the same server as the application). So while the experience is fresh - I'm writing.
Why is it worth migrating from S3
Amazon S3 is an excellent service. Its main advantages are well known: high availability, virtually unlimited scalability, and minimal administration.
But you have to pay for these advantages, and not just for the volume of stored files.
The payment model is built around the use of the service:
- every GET and PUT request;
- every gigabyte of outgoing traffic;
- additional services like CDN.
For most small sites, this goes unnoticed. The problem arises when the load becomes unmanageable.
For example:
- a scraper or hotlinker can generate hundreds of thousands of requests in just a few hours;
- CDN only reduces the load on S3, but is also charged for requests and traffic;
- AWS budget alerts notify about exceeding costs but cannot automatically stop them.
- A coding error. For example, a retry storm in a background image processing task can lead to constant requests to S3. And this is from real experience.
As a result, even a small incident can turn into an unpleasant bill at the end of the month.
If your application is already running on a VPS with a fixed subscription fee, a logical question arises: why pay separately for file storage if the server has enough disk space and network resources?
This is where the idea of a self-hosted S3-compatible storage comes in.
What is S3-compatible storage
The very existence of S3-compatible storages makes such migration relatively simple.
Today, there are many open-source implementations of object storage that support the same API as Amazon S3:
- PUT Object
- GET Object
- ListObjects
- AWS Signature Version 4
- standard Amazon SDKs
For the application, this means that it continues to work through the same client (aws-sdk, fog-aws, or any other library), and only the following change:
- endpoint;
- access key;
- secret key;
- force_path_style parameter (path_style).
The actual code for working with files remains virtually unchanged.
There are several quality self-hosted implementations on the market. They differ in performance, resource requirements, and maturity of the S3 API implementation, but the principle of operation is the same.
That is why the migration does not turn into rewriting half of the application - it is enough to replace the storage address.
Architecture: why writing and reading should be different
The most common mistake during the transition is simply opening the S3 API to the outside and using it simultaneously for writing and reading.
It will work that way.
But it's a bad idea.
Writing files and reading them have completely different security requirements.
Writing
The application uploads the files itself.
It operates within a private network, has access keys, and must see the full S3 API.
None of these ports should be accessible from the Internet.
Reading
Users only need to download already existing files.
For this, it is enough to allow only two HTTP methods:
- GET
- HEAD
Everything else should be blocked before the request reaches the storage.
As a result, the architecture looks approximately like this:
┌────────────────────────────────┐
│ Your server │
│ │
Application ──────▶│ S3-compatible storage │
(private network) │ │
│ │ │
│ ▼ │
│ nginx (read-only proxy) │
└─────────────┬──────────────────┘
│
assets.example.com
GET / HEAD
This approach has several advantages:
- The write API is completely inaccessible from the Internet.
- Visitors can only view files.
- A CDN can be added at any time without changing the application.
- Nginx can independently add caching headers and limit unwanted requests.
In other words, the application and users no longer work with the same interface - each receives only the capabilities they actually need.
Read-only proxy: a small detail that protects the entire storage
Usually, a regular nginx serves as such a proxy. It does not store files and knows nothing about the S3 API - its only task is to safely serve already existing objects.
A typical configuration looks something like this:
server {
listen 80;
server_name _;
# Prevent directory listing.
location ~ /$ {
return 403;
}
location / {
limit_except GET HEAD {
deny all;
}
proxy_pass http://storage:8080/bucket-name/;
proxy_set_header Host $host;
add_header Cache-Control "public, max-age=31536000, immutable";
}
}
The configuration is small, but it is often where errors hide.
Error #1. Open directory listing
If this is missed, a regular request to:
GET /
can return a list of all objects in the bucket.
This does not mean that a malicious actor can delete or upload anything. But they will obtain:
- file names;
- sizes;
- creation dates;
- directory structure.
In fact, you will have given the opportunity to fully index your file storage.
That is why the simplest solution is to completely block requests to directories. The user should know the full path to a specific file. If the path is not available - let them receive a 403 Forbidden.
Error #2. Cache-Control and the always parameter
Another more insidious issue is another problem.
Many nginx configuration examples recommend using:
add_header Cache-Control "public, max-age=31536000, immutable" always;
At first glance, this even seems logical: let all responses be cached equally.
But if a CDN (like Cloudflare) is working in front of the server, the situation can become very unpleasant.
Imagine a typical deployment scenario.
The new HTML has already been published, but the JavaScript file has not yet made it to the new storage.
The first user opens the page.
The origin returns:
404 Not Found
Along with it, nginx adds:
Cache-Control: public, max-age=31536000
The CDN honestly follows the instruction and caches the response for a year.
A few seconds later, the file appears in the storage.
But users still continue to receive 404 from the cache.
From the application's perspective, everything works correctly.
The file exists.
The origin serves it.
And the CDN does not even check this again.
That is why Cache-Control should only be added to successful responses, not to errors.
Public domain without open ports
Another interesting possibility of modern infrastructure is to not open the server to the outside at all.
If a CDN or reverse proxy is working in front of the site, it almost certainly supports outbound tunnels.
The principle of operation is very simple.
Instead of waiting for incoming TCP connections, the server establishes a permanent outbound connection with the provider's network.
Subsequent traffic is transmitted through it.
From the Internet's perspective, it looks as if your server does not exist at all.
Port scanning will show nothing.
No open HTTP.
No HTTPS.
No S3 API.
Only the edge nodes of the CDN are visible externally.
This provides several advantages:
- The IP address of the origin server is hidden;
- The firewall for incoming connections can be completely closed;
- It is significantly more difficult to conduct DDoS attacks directly on the server;
- The storage is not directly accessible even if the IP is known.
For a home server or a small VPS, this is probably the easiest way to gain an additional level of protection almost for free.
Data transfer: where the real adventures begin
On paper, everything looks very simple.
We take rclone.
We create two remotes:
- Amazon S3;
- new S3-compatible storage.
After that, we run:
rclone sync s3:bucket storage:bucket
And wait.
In practice, this is where most surprises begin.
Moreover, they are almost never related to rclone itself.
Usually, problems arise due to small differences between S3 API implementations.
Modern SDKs are sometimes "smarter" than necessary
In recent years, Amazon's client libraries have learned to use new request signing mechanisms.
For example:
- streaming signatures;
- trailer checksums;
- additional integrity checks.
Amazon S3 supports all of this wonderfully.
But not every self-hosted storage does.
As a result, you can end up with a very strange situation.
The access keys are correct.
The time is synchronized.
The permissions are also correct.
And the server responds:
SignatureDoesNotMatch
Because of this, it is easy to start looking for the problem in IAM, keys, or time zones, while the real reason is entirely different.
If you encounter such behavior, try disabling the new signing modes or using the native import tool of the storage itself.
Sometimes this saves hours of searching for a non-existent error.
Content-Type - a small detail that breaks images
Another unexpected problem is MIME types.
Many tools try to determine the file type themselves during copying.
Most often, they analyze the first bytes of the file.
This works great for PNG or JPEG.
But modern formats like AVIF or HEIC are not always recognized.
As a result, the file ends up in the new storage with the following header:
Content-Type: application/octet-stream
The browser no longer knows that this is an image.
In one case, it will offer to download the file.
In another, it will simply refuse to display it.
The best option is to not redefine the MIME type at all, but to transfer it along with the metadata of the original object.
These are the source of truth.
Pre-compressed files - a separate story
If the storage contains JavaScript, CSS, or other text files that were already compressed (gzip or brotli) during upload, it is worth separately checking whether the new storage correctly handles the corresponding HTTP headers.
This primarily concerns:
- Content-Encoding
- Content-Type
Amazon S3 stores these metadata without surprises. But some self-hosted implementations behave less predictably - especially if the files were uploaded by third-party tools.
As a result, the browser may receive unzipped JavaScript, but literally a stream of compressed bytes.
The error looks quite strange.
Instead of the expected JavaScript, the browser console shows something like:
Failed to load module script: Expected a JavaScript module but the server responded with application/octet-stream
or
Unexpected token
At first glance, it seems that the problem is in the frontend build.
In reality, the file exists, but the server serves it with incorrect metadata.
The simplest solution
If a CDN (Cloudflare or similar) is already working in front of your server, pre-compressing files is often not needed at all.
Modern CDNs compress text content during delivery:
- gzip;
- brotli;
- zstd (depending on the client).
Therefore, abandoning pre-compressed files often simplifies the infrastructure and removes an entire class of potential problems.
Not all errors are visible during copying
When a bucket contains several thousand files, migration usually goes smoothly.
But if there are tens or hundreds of thousands, there is a temptation to maximize parallelism.
This is correct.
Otherwise, copying can take many hours.
However, there is another problem.
With a large number of simultaneous requests, some of them will almost inevitably end in error.
The reasons can vary:
- short-term network failure;
- disk subsystem overload;
- temporary service unavailability;
- internal timeouts.
In most cases, these will be tenths or even hundredths of a percent.
Sounds not scary.
But if you are transferring a million files, even 0.05% means hundreds of missing objects.
The most dangerous thing is that some tools may not consider such situations critical and finish without an explicit error.
Because of this, after migration is complete, it is essential to verify the result.
Do not just trust the message:
Transfer completed successfully
but ensure that all files indeed made it.
Checking the number of files is not a formality
After copying is complete, it is worth answering a very simple question:
Is the number of objects in the old and new storage the same?
If the answer is "yes" - great.
If not - do not rush to restart the migration.
It is much more effective to find the missing objects and perform another pass.
Recopying the entire bucket for the sake of a few dozen or hundreds of files usually just wastes time.
That is why the final "re-upload" is an absolutely normal practice even for large companies.
Sometimes even the verification tool lies
There is another surprise that is easy to forget.
On very large buckets, the counting tools themselves may work incorrectly.
For example, they may:
- have an internal limit on the number of records;
- truncate the result after a certain mark;
- cache the list of objects.
In such a situation, it seems that after another synchronization, the number of files has not increased.
Although in reality, new objects have long been present in the storage.
If you see strange statistics, do not panic immediately.
It is better to check a few specific files directly.
For example:
- does the URL open;
- does it return the correct Content-Type;
- does the checksum match.
Very often, it turns out that the problem is not in the migration, but in the tool itself that shows the statistics.
The most dangerous moment - the switch
After a successful copy, it seems that the hardest part is already behind.
In fact, no.
The greatest risk arises precisely at the moment of switching the application to the new storage.
There is often a temptation to do this gradually.
For example:
- first switch reading;
- make sure everything works;
- only then switch writing.
Sounds logical.
But this is one of the most common mistakes.
Why gradual switching is dangerous
In most applications, the file URL is generated from two parts:
- object key;
- base domain of the storage.
If reading and writing start using different storages, a very unpleasant window of inconsistency arises.
Imagine the situation.
A user uploads a new avatar.
The application is already writing it to the new storage.
But the profile page is still generating the URL for the old S3.
The file physically exists.
However, the user sees only:
404 Not Found
A few minutes later, another user opens the same page.
For them, everything works already.
As a result, the problem appears random, is hard to reproduce, and even harder to explain.
The correct sequence of transition
It is much safer to perform the migration in three stages.
1. Fully prepare the new infrastructure
We deploy:
- storage;
- nginx;
- CDN;
- public domain;
- monitoring.
Meanwhile, the application is still working with the old S3.
2. Migrate all data
We perform a full synchronization.
Just before deployment, we run another short synchronization to transfer the files that users uploaded during the initial copy.
This "delta" makes the switch practically painless.
3. One deployment - one change
Only after this does the application simultaneously change:
- the endpoint for writing;
- the endpoint for reading.
Not two separate changes.
Not two different releases.
One atomic deployment.
This way, there is practically no moment when new files can be written to one place and read from another.
Checklist before shutting down the old S3
Before finally abandoning Amazon S3, it is worth going through a small checklist. Most items seem obvious, but they often become the cause of problems after the switch.
- The public domain does not return a list of files. A request to / or to a directory returns 403 Forbidden, not a list of objects.
- Nothing can be written through the public domain. PUT, POST, DELETE requests, and other methods besides GET and HEAD are blocked at the nginx level.
- S3 API is not public. If there is access to it, every request must require correct authentication via AWS Signature V4.
- CDN really caches files. Check the response headers (CF-Cache-Status, Age, or similar) and ensure that repeated requests do not reach the origin.
- Content-Type of all files matches the original. This especially applies to AVIF, WebP, SVG, fonts, and other formats for which an incorrect MIME type can cause issues in the browser.
- Compression headers have not been lost. If pre-compressed files are used, check Content-Encoding.
- All objects have been migrated. The number of files in the old and new storage matches, and random spot checks confirm that files open and return the correct metadata.
- The final synchronization has been completed. Just before the deployment, all files that users managed to upload during the main migration have been transferred.
If all items are completed, it's time to switch the application.
Do not rush to delete the old bucket
After a successful deployment, there is a natural desire to immediately delete the old bucket and stop paying for it.
I would not do that. At least for a month or two.
Amazon S3 is not just a place where files are stored. It is a service with high fault tolerance, replication, and infrastructure that is difficult to replicate on a single VPS.
In contrast, your own storage, especially if it runs on a single server, has a single point of failure.
Even if you regularly create backups, the first few weeks after migration, it is worth keeping the old bucket as additional insurance.
This provides several advantages:
- You can quickly find any file if there is suspicion of an error during migration;
- It is easy to compare metadata or checksums;
- In extreme cases, you can always quickly revert to the previous configuration.
After the new storage has been running for some time without issues, and the backup has been practically verified, the old bucket can be safely deleted.
Is it worth migrating at all?
Like almost any architectural decision, this is not a universal recommendation.
If your service serves millions of users, uses dozens of AWS regions, or requires guaranteed availability, your own storage is unlikely to be a better alternative.
But for small and medium projects, the situation is quite different.
If the application is already running on its own server, and the file storage does not require global replication, switching to a self-hosted solution can provide quite tangible benefits:
- fixed costs instead of charging for each request;
- no bills for outgoing traffic;
- full control over the infrastructure;
- independence from a specific cloud provider.
The most pleasant thing is that thanks to the S3-compatible API, such migration does not require rewriting the application. In most cases, it is enough to change the endpoint and credentials.
At first glance, it seems that migrating file storage is just copying data from one bucket to another.
In reality, the main complexity lies not in copying files.
The most important thing is to build the architecture correctly.
Separate writing from reading.
Do not expose the S3 API to the outside.
Do not allow directory listing.
Correctly configure caching.
Verify file metadata after migration.
And only after that perform one atomic switch of the application.
If everything is done in this sequence, the transition takes significantly less time than it seems at first. Instead, you gain full control over the file storage, predictable costs, and an infrastructure that you can develop independently of the rates of a specific cloud provider.
The most important thing is not to perceive migration as a simple copying of files. It is primarily a change of architecture. And it is precisely how carefully this architecture is thought out that will determine whether users notice the migration at all.
Like it?React
🧵
This post doesn't have any additions from the author yet.
