bava on the Edge

On the edge, I’ve been there
And it’s just as crowded as back home.

Dag Nasty, “La Peñita”

Yesterday I did a little experimenting on the good old bava.blog to test the notion of application delivery networks (ADNs). You probably have heard of Content Delivery Networks (CDNs) wherein static content is delivered via caches all over a service’s global network (most popular being Cloudflare). Well, in this new acronym, beyond the content the whole application itself is cached across the network, so when one (or in my case both) servers driving the bava go down, the site is unaffected, it begins to deliver the application itself through the network. Which means not only high availability, but virtually guaranteed 100% uptime.* I found it hard to believe, and I have been looking into edge computing thanks to Phil Windley’s recent post, but this was my first exploration of the concept.

Our cloud hosting at Reclaim Cloud is driven by the software developed for Jelastic, which was bought by Virtuozzo. It has been something we’ve been pushing pretty hard on with not only apps well beyond the LAMP stack, but also containers and the wonderful work of Docker, which in turn led us to start building a dedicated WordPress service on top of performant, affordable containerized WordPress hosting: ReclaimPress. As I’ve been working through ReclaimPress, I was shown the tool/service Edgeport. Very much positioned as a simplified, easy-to-use Cloudflare competitor, EdgePort was designed as a security-first, cloud-native Web Application Firewall with a global network that delivers applications dynamically, even when the origin servers are off. Their DNS options are an affordable alternative to Cloudflare for similar plans, which has been a key factor for me. To get in the door for enterprise at Cloudflare is somewhere in the ballpark of $3,000 a month (which the condescending Cloudflare sales agent was sure to remind me), whereas all the features we need–many of which are Cloudflare enterprise only—are part of a $199 a month plan at Edgeport. What’s more, I have not seen anything like ADN delivery networks at Cloudflare, so we now have a viable, affordable alternative to Cloudflare which can do even more. That makes me very happy.

I can harness a globally cached network, as well as load balancing fail-over, and the emergency backup of applications being cached and delivered in their entirety from the network (whether or not my servers load), and that is not even including the vast security tools that I have to dig into with Noah in more detail. It seemed like magic, so I spent much of yesterday testing it on this old blog.

I turned off both servers in the failover setup at 10:59 UTC and then powered them back on at 19:48, so just under 9 hours of downtime that did not stope a single page or post from working cleanly on my site.

Image of Log for when the servers were turned off and then back on

Log for when the servers were turned off and then back on

I had Antonella try and comment and that was not successful, and never thought to try logging into /wp-admin area, given it would seem impossible, but maybe not?  Will return to that, but perhaps comments and posting do work in an ADN?†

Regardless, it was fun to occasionally search for blog posts that I hadn’t read in years, and see them load without issue, even though both servers were down.

This comes at an amazing time at Reclaim when we’re going into our second year of stable, solid .edu hosting for a number of schools, and adding this possibility for not only guaranteed uptime, but increased vigilance and next-level cloud-based security is pretty thrilling. I really want to get out on the presentation trail again and talk this through because more and more these leaps in infrastructure are something we have been just able to almost keep up with, but this one almost feels like we are not only well-positioned to offer it, but maybe even early to the party.

Reclaim4life, small and limber is beautiful!

_________________________________________________
*With the caveat that is an imagined Shangra-la if you push hard enough on the idea.

†Turns out they cannot make the database writable in the ADN, so it is read only. They mentioned it is technically possible, but not legally—which makes sense when you think about it in terms of security and spoofing, and then there is the whole issue of syncing back changes. It might make sense, if only for practical purposes, to keep everything write-only during any extended downtime.

Posted in reclaim, Reclaim Cloud | Tagged , , , , , | 1 Comment

The Perfect Gameroom

If you were a New York Yankees baseball fan in 1998, one of the highlights of an all-around amazing year is the unlikely feat of everyman David Wells pitching a perfect game one sunny May afternoon. The excitement was electric, the whole world seemed to be re-framed by this one person’s specific achievement. The idea that for that one day, those 2 hours and forty minutes, his pitching was perfect. Everything worked, everything in its place—magic.

David Wells being carried on his teammates shoulders after perfect game in 1998.

While I may have David Wells’ body frame, I have none of his remarkable athletic abilities; so this is a tale of perseverance and commitment (rather than skill) over 2 and a half years to make sure every single game in my personal arcade collection (the bavacade) works perfectly. Well, the day has come, and I have evidence in the form of video below, but the real point is that for a few hours (days even? -never weeks 🙂 ) this collection can be perfect, every arcade cabinet in the bava manse works as they did back in the early 80s.

Today is a good day, even I can be a hero, just for one day!

Posted in bavacade, bavarcade | Tagged , | 2 Comments

bavacade repair log 11-12-2023

Well, I think the time has come where all 28 games here in the bavacade are working 100%. There are still some small things to do, but if everything goes as planned in the walk-through today I’ll have 28 games running and working seamlessly in just a few hours. But before I mark that momentous occasion with it’s own post, I wanted to track what I have done since my last repair log update.

Elevator Action is stencilled, re-assembled, and working beautifully. With the new control panel overlay and slick paint job, this game is aces right now. I just have to do some touch-up paint on the rear door and get a few scrap pieces of wood to build something the AC input unit can hang on. I love this game, and I love that it is now mint and a pillar of the collection highlighted in the Foyer 8*

Elevator Action

Stargate continues to allude me, I had the resetting problem return again, which still threatens the perfect game room. Last night I swapped the switching power supply for an original unit I had, and that game seems to be playing now. We’ll see if that is the one that spoils today’s fun.

I had Cheyenne back up and running, but when I turned it on last weekend the power would not go on. I pulled the power supply out and tested everything, but all seemed to work, it was just the interlock switch that was not working—>I should have know then. Anyway, after pulling the game apart I realized the kill switch in the coin door was open, causing no power to flow. A rookie mistake that cost me hours.

Yie Ar Kung fu custom side art

The Yie-Ar Kung-fu artwork is amazing. I cut that into slivers and applied that last week, and a few places the stickers were not perfect, but well hidden and overall it is pretty amazing. All credit goes to Bryan Mathers for creating the art and Ricky making it part of his high school project and printing the art out and working with me to make it all fit.

Yie Ar Kung fu custom side art

At this point I just need to get the glass bezel and glass marquee printed, and then figure out the best way to approach the control panel print-out.

Yie Ar Kung fu custom side art

Despite all my love and attention, the freaking Yie-Ar Kung-Fu board started having some character pixelation, which has happened before. I swapped boards so that it’s now working well. But I wonder if something else is up because Mike (who fixes these boards) assured me it is not a board issue, and power supply checks out, so this might need a re-worked edge connector. But right now it works!!!

Apart from that, the original power supply for Scramble was fixed, so that is now installed alongside the switching power supply—both working. With that working, I decided to get the Super Cobra power supply fixed as well, so the next thing I might try and do is go through all the power supplies and fix those, and then try converting more games to 220V.

I think that is pretty much it, after two and a half years I have a fully operational arcade, at least for today, and that’s enough!

_____________________________

*The eight games I have in the foyer that all shine like new: Pac-man, Donkey Kong Jr, Galaxian, Dig Dug, Super Cobra, Elevator Action, Joust, and Defender.

Posted in bavacade, bavarcade | Tagged , , , , , , | Leave a comment

Future of What or, Abstractions of Self in an Online Oblivion

Image of Unwounds studio album Future of White, a black and white modernist affair with cubist illustrated building and stark blacn and white background

Unwound’s studio album Future of What, a modernist affair with cubist illustrated building and stark black and white background

Just finished a vinylcast on #ds106radio of Unwound’s 1995 album Future of What while I came up with an idea for a possible talk at OER24 in Cork. I’m going regardless, and tomorrow is the deadline so I might be remiss, but I would love to be able to present something semi-intelligible at my very favorite conference. I’m not sure this is it, but it is the beginning of something for sure…

This presentation will be a meditation on the changing nature of one’s online self when corporate social media has all but colonized the web. The moment wherein social media companies rise to the level of geopolitical powerhouses with seemingly limitless resources viz-a-viz billions and billions of individualized data points has arrived. The question many are struggling with in the aftermath is whether resistance is futile and assimilation into the pod-cast-people inevitable. 

All of this is further complicated by the rise of the machines in the form of artificial intelligence hype at the precise moment when social media diaspora and online existential crises are at peak levels not only for higher education, but for any soul interested in being social online in 2024.

The digital storytelling course ds106 is taking all these questions head-on during the Spring 2024 semester, across three universities and an open contingent, the attempt to usurp the pervading sense of powerlessness underlined by a dreadful seriousness with a healthy, sustained dose of  playful collegiality wherein we learn to interrogate the tools being used to access and exploit the immense pools of data collected over the previous twenty years of the social media free-for-all. The future of what is what this session will be.

Posted in digital storytelling, OER24, presentations | Tagged | Leave a comment

Macaulay Migrations, Random Litespeed Issues, and S3 Media Offloads – Oh My!

[Lessons learned: never do a victory lap on your blog before the migration is in the bag.]

It’s been well over a month in the making, and in many ways it started my descent into the Mouth of Madness that I’m currently ascending from—so let’s revisit, shall we.

In the Mouth of Madness GIF

Reclaim Hosting is in migration mode, we’re working to migrate larger WordPress Multisite instances and cPanel servers to newer infrastructure as we stare down the barrel of the upcoming CentOS 7 end of life. I took the occasion of getting ReclaimPress up and running to take on one of our more complex WordPress Multisite instances, Macaulay’s venerable Eportfolios, and moving it into a containerized WP instance.

What’s more, part of the complexity of this site was that the MySQL database was running in a separate Digital Ocean droplet given how resource intensive it was when Tim migrated it 6 or 7 years ago. So, this migration would be moving it off a cPanel server into its own container, and also consolidating the database into the same container. We needed to make sure there would no be performance issues, which luckily there were not, and also ensure that w gave ourselves enough time for the migration given it took 17+ hours to drop the database on the outdated Ubuntu server—making this migration particularly onerous.

The other piece we wanted to solve was trying to get the over 700 GBs of media off into an S3 bucket to save space on the server. As we run more and more in the cloud, offloading sites with a large amount of media (200+ GB ) becomes more an more important given media eats up space on our dedicated, bare metal cloud servers (the cloud is just another server, it turns out). So, Eportfolios was going to be our first experiment with doing this not only in a larger WordPress Multisite besides ds106, but also running the media through AWS’s S3 and content delivery network (CDN) Cloudfront, given we’ve only ever done this previously with Cloudflare.

So, it took me some time, but I did get the consolidation of the database and the migration from cPanel and ReclaimPress figured out. What’s more, once I did the site was running lightening fast in dev, so that piece seemed all set. A major issue I ran into was getting a PHP 7.4 LLSMP* container running easily on ReclaimPress, the other was timing the MySQL dump from the stand-alone MySQL Droplet. But once they were under control, the site ran cleanly and was quite fast, even with the consolidated database.†

So once all was set and the switch to the new environment happened the site was loading quickly, the only issue was all the media was broken. WTF?! Oh, probably just the upload media path…no! Wait, it has to be permissions…no!! OK, then really it HAS to be a .htaccess problem…NO!!! You get the idea, for about 12 hours Macaulay was not resolving images and media files while we frantically dug to figure out what the hell was happening. Luckily, Chris came in for the assist and did what was some next-level forensic analysis on one of the images that were not loading to scan for any differences between the original that did load, and the migrated image. Turns out there was a single character added to the beginning of every media file’s hex code that was corrupting anything trying to load.

Here is hex code of an image that was loading cleanly on the old server, and we confirmed it’s the same as the image on the new server as well, so no issue with the image itself:

Image of Hex code of the uncorrupted image file that was from the old server before migration

Hex code of the uncorrupted image file that was from the old server before migration

But when the image loaded the characters 0A were added, essentially corrupting this file, and every other one.

Image of Hex code of the additional characters added the characters 0A to beginning of all media that was corrupting files loading on eportfolios

Hex code of the additional characters added the characters 0A to beginning of all media that was corrupting files loading on eportfolios

Major kudos to Chris Blankenship for figuring this out using a image hex decoder that ultimately allowed us to put in a ticket to Litespeed, given the corruption was happening at the level of the web server, not the image. Turns out the file wp-includes/ms-files.php was adding this extra character, and once we resolved that the entire site was working as expected. Damn that was rough, I was so convinced I had nailed that migration, what did I say about premature victory laps?

With that resolved and the site loading as expected, other priorities quickly took over, but we had pushed 700+ GB of media to an S3 bucket on AWS. We were not loading media from the S3 bucket yet, but that was the plan to free up almost a terabyte of space on the server. Just this week we returned to this part of the migration in earnest, and this was in many ways a first for our team to experiment with mapping a domain to AWS’s Cloudfront so that all media would run off a Macaulay branded subdomain: files.eportfolios.macaulay.cuny.edu. This required copying everything in the wp-content/blogs.dir directory to a bucket named files.eportfolios.macaulay.cuny.edu. After that, we needed to create a distribution network on Cloudfront that used files.eportfolios.macaulay.cuny.edu as an alternate domain. We also needed to get an SSL certificate through Cloudfront using a CNAME certificate—which was set to expire after 72 hours if not activated, so a bit tricky to coordinate.

Image of WP Offload Media Interface

WP Offload Media Interface being delivered through Cloudfront

The other big piece here is using WP Offload Media to make this all work, and that has been clutch. I  wrote about that plugin while using it to offload media for both this blog and ds106.us, but this is the first time we did it for this many files and using Cloudfront, so definitely a brave new world for us. There may be a bit of clean-up this morning, but as of now—and a bigger proof-of-concept for offloading media for much larger, media intensive sites—this is a huge win for us.
_________________________________________

*Getting a PHP 7.4 environment to run on ReclaimPress is an unnecessarily round about process right now given you are forced to import from an older container manifest, namely https://raw.githubusercontent.com/jelastic-jps/wordpress/v2.2.0/manifest.yml

†Dropping the database in the new environment only took 15 minutes (as opposed to 17 hours) to dump in the new environment—that alone was a huge win.

Posted in AWS, plugins, s3, WordPress, wordpress multi-user | Tagged , , , , , , , , | 1 Comment

The Return of Unwound

image of me In the lobby at the Wiltern ready to rock with Unwound

In the lobby at the Wiltern ready to rock with Unwound

What was wound? Back in February I attended an Unwound show at the Wiltern in the inimitable Los Angeles. LA is a special city to me, given I spent some of my formative years living there and attending UCLA #4life. The things that make you you are varied and complex, but there is definitely a big part of me I take from my time in LA from 1990 to 1997. One of them is Unwound, they were the soundtrack of most of my time there, particularly Fake Train, New Plastic Ideas (1994), and The Future of What (1995). I saw them more than once at Jabberjaw in 1994 or 1995 and it was magical, I wrote about that experience back in 2010 right here on the bava.blog.

Los Angeles, Jabberjaw, Unwound

Hell, I even travelled from NYC to Seattle in 2002 to catch their penultimate show in Seattle, alos wrote about that here.

Summer of Love: Unwound

In fact, we had one of Reclaim Hosting’s servers named after this band in honor of THE post-punk band of the 90s. And guess what, I wrote about that too!

What Was Wound

Anyway, you get the idea, I’m a fan, and have been for 30 years now. So, the idea of seeing them for the first time in 20 years was truly exciting, and I was joined by some amazing company and this show would be just one small highlight of a much larger and rewarding adventure. Nothing like going back to the land of the formative years triumphant.  LA taught me how to escape the strictures of the past, and it remains a wonderful teacher.

The show was what you would expect from Unwound, honoring the memory of deceased founding member Vern Rumsey with a low key, heartfelt shout out. In fact, the show was in many ways how I remember them, subdued while rocking, and Sara Lund leading from behind the drums, and I was in heaven. There was even a moment wherein Sara engaged with the crowd asking folks how many of us saw them at Jabberjaw, my hoots and hand waving were instantaneous, along with a few other aged souls. She followed that with how many of you were not born in the mid 90s, to which a much louder, vigorous response was raised. Vindicated, Sara said this show is for all of you (suggesting the new generations of Unwound fans), as a result I was feeling a bit demoralized. I mean I could name there songs, I even tried to re-create the set list from memory the following day as traveling out to the desert to escape even more. But alas, Lund’s comment stuck in my craw. I mean, not for nothing, but how many of the other hippies in attendance traveled all the way LA from Italy to see them live? How about a little something for the effort, Lund and Trosper!!! Oh, how sharper than a serpent’s tooth is a reuniting band’s ingratitude.

Image of Unwound's setlist written from memory of the February 15, 2023 chow at the Wiltern

Image of Unwound’s setlist written from memory of the February 15, 2023 chow at the Wiltern

Above is the set list I put together from memory of songs played (although not in chronological order) while driving out to the desert the following day. And Mikhail tuned me into the fact there is a site that does this for you, so here is the “official” setlist that I imagine is in chronological order.

Image of setlist.fm entry for Unwound at the Wiltern on February 15, 2023

setlist.fm entry for Unwound at the Wiltern on February 15, 2023

As I said already, the Unwound show was only one part of a longer west coast tour with co-pilot Brian Lamb that was truly amazing, and the thing I came away with was that Unwound did the 20 year re-union tour proud with the modesty, dignity, and a stormy-minded intensity they rocked during their initial run, and the fact they’re looking forward to the future, without being hung up on the past filled with old jackasses like me represents another example of a band that knows who they are where they’re going. May we all take a page from their book, even if it hurts a bit for the frail egos in the audience. I’m glad I saw them again, and here is to the next time I rock out to their music live.

Posted in music, vinylcast | Tagged , , , | Leave a comment

Building Oblivion University in the Aftermath

Well, I think it’s finally time to start blogging about the ds106 course that’s likely to run across at least three universities in the Spring of 2024. The course, in short, is going to focus on digital storytelling in the wake of AI, the aftermath—if you will. I’ll be teaching alongside such stalwarts as Martha Burtis, Paul Bond, and Michael Branson Smith (MBS), which is reason enough to do it. The narrative for the course is currently unfolding in our collective minds, but when MBS mentioned framing it as “the Aftermath” based on some of the plot lines Martha and Paul have been working through, I knew that was going to be the name: ai106 – the aftermath (the ai106 was Martha’s stroke of genius). I’ll spare details on the course given that is very much a work in progress, but in broad strokes there is a moment after some imagined event, and that becomes ground zero for the rise of the machine. There will be multiple plots and subplots, many of which the students will create, but there will be this looming figure of Dr. Oblivion who has arisen in the aftermath and has started an AI-driven university. What’s more, he will be actively recruiting students to leave their current institutions and join the “the one true university,” the long overdue Oblivion U—where truth can finally be measured.

I think you can see the possibilities there, MBS has already trained the AI on Dr. Oblivion, so that is all set, we just have to train him to teach university between now and January—should be a breeze 🙂 The fun part for me is building out the entire infrastructure for this brave new world of online universities. The idea is to take the elements that made up our Open Media Ecosystem  last December, and bring them all together using Mastodon and Mattermost to essentially build out an entirely open source platform that not only allows us to own the publishing, as WordPress so beautifully did, but also the comms via social media where Mastodon replaces Twitter; Mattermost substitutes Slack; PeerTube premieres over YouTube; PixelFed stands in for Instagram; Azuracast for, well, Azuracast; etc. You get the idea, I’m sure, it will be a full blown university in the open source cloud driven by everyone’s favorite VHS-born professor. I love the idea of it being “the aftermath” given ds106 has been in a bit of a slumber and the recent rise of the machines has signaled a shift in the rhetoric of a kind of before and after when it comes to AI. So it only makes sense to mobilize the rebellion to explore that tension, here’s to hoping so many of you beautiful machines out there join us!

Posted in digital storytelling | Tagged , , , , , | 7 Comments

bava.studio

So I was talking with Tim a couple of weeks back, and I was bemoaning the difficulties of getting an arcade up and running in Italy. There are some major hurdles, not least of which is the original security certificate the game must have to be allowed to be publicly accessible for commerce.* The worst part is I have no desire to re-create a proper arcade like Reclaim Arcade, I mean why would I? -it was perfect the first time 🙂  I just want the 30 classic video game cabinets I’ve spent the better part of two years minting out accessible to someone other than myself.† So, back to Tim, he said in his matter of fact way “why don’t you just get a space and start building without too much overhead, like we did in the CoWork space back in 2016.”

Eureka! He was absolutely right, I was overthinking this whole thing, I just needed to do it. And once I broke through the logjam, the ideas started flowing, for example, why just a place for games? Why not a retro cultural center? Or a space to experiment with old and new AV equipment? Maybe a little 3D printing?

And then it hit me, Given I have no real desire to make this a commercial space, why not just turn the whole thing into a kind of stream of consciousness living exhibit, wherein I use the public-facing spaces (the shop windows) as a kind of looking glass into various cultural vignettes. Like staging a scene from one of my favorite horror movies, and the next week/month after that highlight a few Williams arcade games (and make them remotely playable even?).  How about taking the hundreds of Smurfs I collected when I was a kid and re-create the Smurf (or Puffo in Italian) village. What if I use the exposition space to recreate a wall of 80s VHS tapes for another installation and on and on and on.

All the while the exhibits will link to a site (bava.studio) that will provide in-depth information, links, context, and more. And if it catches on, make calls out to the community to submit ideas for the next window that they can build and get featured in the window for a week or so. You get the idea, right? —it’s basically this blog, the veritable bavatuesdays, but in storefront form.

That’s right, hippies, this blog is about to get a physical address. In fact, this afternoon we found a modest 600 square foot space downtown (right across from my favorite Pugliese bakery) where it can take shape. It’s kind of a ridiculous concept, but as downtown commerce increasingly goes online, the city center becomes less and less viable as a shopping mall. So, what if all those spaces get reclaimed and transformed into cabinets of curiosities through which we can connect with another through spaces of playfulness and creativity?

I’m sure there is nothing new under the sun, and this has been done before, but I’m pretty excited about giving it a go and seeing what comes of it. For me, the real allure is re-directing my energy refurbishing the arcade games into creating scenes and sets for the Trentini to see and say to themselves, “WTF is that?!” 🙂

________________________________________

*This mythical certificate is near on impossible to get given many of these games were not even in Italy when they were created, and most of the original security certificates that did exist for these games in Italy are long gone now.

†Something I learned from the Living Room Console was that an exhibit like this needs to be inviting and playable by the public.

Posted in art, bavastudio, bavatuesdays | Tagged , , | 2 Comments

Containerizing Mastodon on Reclaim Cloud

Taylor and I did a stream yesterday wherein we (royal) set about taking one of my experimental Mastodon instances (bava.social) and moving it from a Debian VPS into a Docker container. We got the bug after a recent open source trifecta update I posted about, and I am glad we did. Long story short, the genius of Docker is how it makes hosting more complex server setups like Mastodon simple (as with Azuracast and PeerTube), which in turn will make running and maintaining these open source systems that much easier, hopefully leading to wider adoption. There is still the question of cost, but I have ideas there as well, but that is another post.

Anyway, the 45 minute stream was a roaring success! We were able to get bava.social off the VPS and onto the 1-click installed Docker instance running on Reclaim Cloud. It must have been a proud moment for Taylor to see all his work on that installer and increased familiarity with Docker come together in a seamless moment of everything just works. Anyway, if you are into this kind of stuff, the stream has been recorded and I embedded it above.

Posted in docker, Mastodon, reclaim, Reclaim Cloud | Tagged , , , , , , | Leave a comment

bavacade repair log 10-25-2023

Things are definitely coming along, this weekend I got the K4600 chassis for Challenger working, which is ostensibly my last big hurdle to perfection…but things might change before I rebuild Elevator Action.

For example, the Stargate game started resetting again, even with the new FPGA board, so I swapped in the extra Joust main PCB board, and it worked for a bit…then the game would go to white screen of death whenever I pressed the smart bomb button. I then got the idea to unplug the main board from both the ROM and Interface boards, and that seems to have fixed the issue, and the FPGA now works seamlessly.

Painted the front and right-side of Elevator Action, so the only thing left is some touch-up paint and a re-build today.

Alberto applied the new overlay to the Elevator Action control panel, and that just came back yesterday, so I should have Elevator Action rebuilt today—and maybe even a perfect arcade?

Some additional tedium: K4600 in Challenger originally meant for Venture, but when it failed had to use Challenger K4600 to get Venture working. In terms of the fix, I really just disassembled and re-assembled the chassis. The anode suction cup seems to come off given the metal piece holding it in is not tight enough, worth looking at and maybe swapping

To test Stargate I took the main Joust board backup and swapped it with the main Stargate board, it worked, but when main board was still connected to ROM and interface boards we still got issues, I unplugged main board from PCB board, and the restart has not come back, and I think the issue is between the ROM board and main board, and given I don’t need the ROM board will keep this connector unplugged.

Posted in bavacade, bavarcade, video games | Tagged , , , , , , , , | 2 Comments