In my work to get familiar with Reclaim Cloud I have been taking on projects that I am fairly familiar. Such as migrating this blog, ds106, and ds106.club. It’s been educational, and my last migration project is now almost finished as well. I wanted to migrate both my AWS S3 and Digital Ocean Spaces accounts over to the Reclaim Cloud.
But before I go too much further, it might help to understand what I use S3 and Spaces for. Back in 2014 or I started playing with S3cmd line as part of the process of moving bavatuesdays over to Reclaim Hosting—which is crazy to think about. The use in 2014 was pretty specific, using the Amazon S3 command line to upload a very large file so Tim could help me with the migration of my blog.
Since then I have used S3 not only for random storage, but also as a place to also upload a copy of any media I add to the bava using the WP Offload Media Lite for Amazon S3, DigitalOcean Spaces, and Google Cloud Storage plugin. As the plugin notes, Amazon’s S3 is now just one of many players, such as Google and Digital Ocean. Cloud storage, which is referred to as Object-based Storage, is different from block and/or file-based storage in that it uses a flat structure made up of objects that are related to one another through metadata rather than file hierarchy or block structure. So, rather than controlling access and URIs through file structures or separate parts, you can control access to a single file directly through metadata, something that is crucial for container stack architecture to work given each layer of the stack is usually abstracted from the underlying server.*
So, back to my use-case that I’m reproducing in Reclaim Cloud, I have the WP Offload Media Lite plugin on my blog that connects to S3 and Spaces and uploads all media from my blog there. This is useful for have a backup across multiple regions, hooking media into a content delivery network, or simply separating uploaded content and media from the core application files, which arguable make a move to a different application that much easier. So, as I was imagining a solution Tim quickly pointed me to MinIO, which is open source object-storage software like S3 or Spaces that we just so happen to have a one-click installer for on Reclaim Cloud, you see where this is going now?
So, I installed an instance of MinIO on our Canadian data center (that’s right, Canada!) and called it bavamedia.
It took a few moment to install, and I was up and running with an object-storage solution, now I needed to see if it worked cleanly with s3 commands, which it does. I had already installed the S3cmd tool on my Mac (the guide is quite good for getting it running), so I’m now able to push files from my desktop up to MinIO. This means I could download my media uploads from AWS and Spaces, and uploads them to my Minio Instance, which I did. Once the directories were downloaded, I needed to upload/sync them to MinIO which means using S3 commands like the following:
s3cmd sync --skip-existing ./ s3://files/
This command syncs all files within the current directory (./) on my desktop with the /files bucket on MinIO. So, in my files bucket I am either putting or syncing everything in /wp-content which will then be the bucket I point the WP Offload Media Lite plugin so that all future uploads to bavatuesdays go to this bucket. “But does this plugin support MinIO?” you naturally ask? “It is obvious from the title we can use S3, Spaces, and Google Cloud, but what about MinIO?” Absolutely, it uses the S3 command language, so it works cleanly, you just need to add an additional plugin, WP Offload Media Tweaks, and edit the amazon-s3-and-cloudfront-tweaks.php file with specifics from you MinIO setup. In fact, Delicious Brains, the developers of the plugin, have an excellent guide that takes you through using their plugin with MinIO. In fact, Tim was also playing with MinIO, and effectively realized that we can provide folks hosting Omeka (or Omeka-S) the ability to integrate through MinIO, much like they could with S3 previously.
If you do use the WP Offload Media Lite, I have found turning off the object versioning helped me keep the file structure identical between my blog and my off-site files on MinIO (the object versioning adds another directory layer), but there is a sacrifice of the the extra layer of metadata for file structure consistency, which highlights that my thinking is still straddling the old and new in terms of my comprehension of object-storage.
Below are areas that highlight my filters for connecting to the MinIO object-storage through the Amazon S3 and Cloudfront Tweaks plugin linked above:
And the following
add_filter line was already uncommented and ready to go. I added it in the plugin code above this line and kept getting fatal errors when trying to install the plugin, only to realize it was already in there and ready to go.
After I did this MinIO works as a replacement for Spaces and S3 in terms of offloading media from my blog, but now it is time to get serious about exploring the possibilities that this flat file, metadata rich option provides me well beyond WordPress media.
*I’m still working my way through this conceptually, and this post is part of that, so any and all pointers, metaphors and explanations of these differences is always appreciated.
Jim, I have enjoyed following your journey lately exploring different hosting options. I was particularly interested in the way you store your images outside of your blog using WP Offload Media Lite plugin. I am wondering if this would be an answer for having two sites and continually being frustrated with my workflow in regards to where images are stored. I have taken to using the External Featured Image plugin as a workaround, but am wondering if the same external media folder could actually be attached to both sites?
I imagine that whatever I do I am probably going to have to go through a bit of a clean up to get it to that stage.
Funny, I am working through something similar right now where I can have one networked file system for several sites. I have not cracked the nut, but I believe it is possible, and will try and work on a prototype here this week. It is fun to explore the possibilities of containers and Docker for this stuff. What I am learning is the idea of abstracting out the stack makes symlinking a common uploads file from several applications definitely possible. Is this for two WordPress sites? If so I ca work on a demo, and after that it might be something you can explore yourself in the open beta of the Reclaim Cloud 🙂