Train wreck

For all of you out there who deeply question the “system” model, and the idea of hosting this stuff for an institution and making it work on a larger scale. Today I agree with you,  for we’ve had nothing but consistent downtime with UMW Blogs for the last four or five day, and while I often blog about how great UMW Blogs is (I’d link to it, but it is probably down so what’s the use), today it sucks.  And it also makes me think that despite my constant notions of creating a space like this for innovation, community, and experiments—it inevitably turns into a huge albatross around our necks.

What’s causing it?  Well, if it would stay up long enough we might figure it out. It could be some constant call to wp-cron.php that we can’t precisely locate (though disabling cron seems not to help).  Could be that we are being spam attacked every ten minutes-though there’s no certainty there either.  Could be our real traffic is far too heavy and regular for our server. We’ll be upgrading to a bigger server shortly, and I am praying this will keep us up so that the professors and students who have come to count on UMW Blogs don’t start jumping ship en masse, but at the same time maybe they should. Maybe it is just another enterprise system after all, and what we’ve all labored over for two years has seen it’s last innovative breath. I mean who cares, here comes Google Wave and Twitter and all the other newest and bestest tools that we have to starting imagining pedantically.  I hate the whole whole field these days, so I’m gonna return to my nostalgia and Sears Catalog and forget about my day job for as long as I can. Edtech you suck!!!

Image credit: rcstanley’s “Train Wreck – Prob early 1900’s”

This entry was posted in UMW Blogs, wordpress multi-user, wpmu and tagged , , , . Bookmark the permalink.

11 Responses to Train wreck

  1. peter naegele says:

    I feel for ya…just put up a static page stating that it’s bava appreciation week and you took the site off line to make people appreciate it!

  2. Cole says:

    I feel for you … blogs at psu has been in similar waters the last couple of weeks — its running, but it is sucking the innovative life out of my team. I’ve been asking why in the world are we running this and not just working to federate identity into existing systems — you know, people who are paid to actually think about keeping stuff running at scale. I met the TypePad team and they are wicked smart and their real business relies on them keeping a massive multi-user blogging environment going. I don’t know the wordpress.com team, but I suspect they know what they are doing as well.

    One thing to remember — you proved that an environment that is open is better than one that is closed. Does it have to be open on your campus? This is the syndication bus — let them go to whatever service they want and just come back and drop the feed. Running any of this stuff, once it approaches scale, is no fun. Live well and have fun, Bava! We are counting on you to blaze the next path that we can walk down. I am betting you’ll get it figured out, but when it is back to running perfectly keep thinking and writing about what we should all be doing next. Hang in!

  3. Scott says:

    Hey Jim,

    There are a few things that you can do to get a quick boost to your server. Install PHP APC so the php can be cached saves going through the interpreter each time this will add give 30% performance gain to to the server. Also MySQL caching that will also save a lot of CPU cycles as well.

    This is a good post on the wp-cron.php issue (If you haven’t seen it already)

    http://trinity777.wordpress.com/2008/10/28/wordpress-26-the-issue-of-wp-cronphp/

    some good solutions in there.

  4. Andre Malan says:

    Maybe this is a sign that the pieces are just getting too big. What started out as a “small piece to loosely join” has slowly started to grow.

    Scott was just tweeting about using Tornado or Django to do aggregation instead of using FeedWordPress. I think that is a great idea. Pull that piece off (we would have to come up with an alternative to “add to FeedWordPress”, but that should be easy) so that it does risk the other pieces.

    UMW blogs has too many users. I would say that most of them should be on WordPress.com. It makes sense for most of the use cases. Of course there are some things that only UMW blogs can do, but those are the only things it should be doing.

    I think if you trim the fat you will find that everything will start to work again and you guys can get back to that awesome innovation that you do so well.

  5. Reverend says:

    @Peter,
    I like that, i like it a lot.

    @Cole,
    Yeah, and your issues are probavly far greater than ours. We actually got everything squared today, but the pressure you feel when hundreds, if not thousands, of people are depending on the service for teaching and learning. It puts a a big pit right in the center of your stomach. On a more positive note, i loved your recent posts about sharing quotes via a quick tag process. Kinda like a twitter for miscellaneous quotes and images, etc around the web, I like that a lot. I’ll have to see it in action to get a sense of the interface.

    @Scott,
    That post on wp-cache confirms my worst fears, it is basically doubling the traffic on our server with every click, which means we had to jump servers. We went to a much more robust system last night in a emergncy surgical opreation, looks like everything is sailing along today, which changes my outlook for the moment. Crazy thing is we had PHP APC running, but had some issues with the MySQLcaching given the multi-db setup we were running.

    Cast Iron Coding, who takes care of our server setup, was bending over backwards all week, and finally said, “listen, it is time to buck up and get a bigger server, because even once you contain this stuff, it is still extremely resource intensive.” And so far that seems to be the case, it is humming for now, but I also found an infinite loop coming out of a FeedWordPress call to a category that kept republishing that same post over and over again on ever new call to the sitewide category feed. It was bogging stuff down majorly, as were a few other things. The syndication bus ain’t easy or free, and I was about to hop off at the nearest bus stop last night.

    @Andre,
    Yeah, the system creep is occurring, and I am fascinated by the idea of doing the syndication through Django. We have an alumni who is big into that application and I would love to see what that is all about.

    And, I agree with you, we need to push away from UMW Blogs for everything, and start focusing on what it can do, it IS time to trim the fat, I agree. The syndication bus does need to be separate. As usual, you are right!

  6. Matt says:

    WTF? What happened to the Jim Groom I know? That Jim Groom would never let a bit of server downtime leave him doubting his entire project. Did @fakejimgroom take over the bava or something?

    If wordpress.com can host hundreds of thousands of blogs, so can any installation of WPMu, provided it has enough server bandwidth. The one thing that WP.com does that you might want to look into is trimming the fat on your PLUGINS — notice that wp.com offers many fewer options to its users. We (and here I mean we edtech people) need to think about the wisdom of activating plugins simply because they do something cool, without regard to how resource-intensive they are or how well they work. We need (and, actually WORDPRESS the company needs) a better vetting system than user ratings.

    But knowing you have some work to do doesn’t mean that you throw the entire project under the (syndication) bus!!! Remember: the reason to keep umwblogs going is that it gives users control over their own content, and third-party apps often put that content out of user control. Yes, that would be a relief for a weary site admin, but it is also potentially a devil’s bargain. Just ask (former) mag.nolia users.

    So: go chill out, get a massage, go for a swim, whatever you need to do to relax. And then, bring UMWBlogs back.

    Jim, you can rebuild it. You have the technology. You have the capability to make the world’s first bionic website. UMWBlogs will be that website. Better than it was before. Better, stronger, faster.

  7. Brad says:

    As Cole mentioned, this semester blogs@psu has had its own growing pains. It is a cycle with me too, when things are working, there is no limit to what it seems the system can do or be used for, when things aren’t working so well, start looking for tools other than blogs.

    As more and more people use and count on umwblogs or blogs@psu, the systems start to become a little less agile, and the speed of innovation slows. It sucks. Maybe it is not inevitable though, if people are willing to stay agile and bend with the system. Maybe we are moving to an era of more agile users.

    I am really curious about the comments of trimming the fat from UMWblogs, and the fact that a lot of the stuff should be on wordpress.com. At PSU it seems we are still increasing the variety of ways the platform is being used. Really it is a platform for a growing number of modes of communication, not just blogs. Hopefully this is the right path. What are you doing that should be on wordpress.com? We have our own .com discussions here are PSU, I am wondering if they are the similar to yours.

  8. Luke says:

    One thing that came out of a meeting I had with Jane Wells yesterday is that Automattic (not WordPress… there is no company, “WordPress”) is aware of the inadequacy of current plugin production and vetting processes, and has a really good plan to address it that both moves us in a more thoughtful direction in our use of plugins and promotes open source goals. Matt, I can go into more detail over pies on Thursday (only because I don’t know what’s been made public and whats to be announced).

    I also agree with the argument that scale and usage aren’t the problems; the issue is having the ability to support and adapt to them. That’s not rocket science, but it does take some attention. This has me incredibly concerned about B@B, as we’re essentially a one man show with no sys admin to speak of, and we’re quickly approaching UMW’s scale, although with less feeding going on (I’ve purposefully kept sitewide tagging off the system because of my load concerns, and we only have a few projects using FWP). I’m doing all I can to make noise and let folks know that our current setup can’t continue; in the meantime, I’m controlling plugin usage like a fascist and being very careful about what I add.

    The decentralized syndication bus simply won’t work in our situation, where I’m battling several ingrained cultures that make it difficult to realize (BlackBoarditis, a resilient belief that edtech is client services, a commuter campus, a real digital divide, perpetual under support for innovation). I need a system that I can watch because without it, I just won’t be able to help members of our community find the connections that can make their work richer. Hell… we’re still dealing with a community where 90% of folks go blank when you mention the word “feed.” That has real consequences on my work, since now that we’ve got this thing going I need the community to show me where to take it. A true syndication bus is a pipe dream for us. Maybe we can have a syndication crosstown bus?

    None of these problems of scale are insurmountable, and conversations and sharing like this is the best path towards optimal setups for each of our projects. I hear your concern, Jim, about the “system creep” with putting all of this responsibility on a singe platform, and share your desire to get other tools in the toolbox. Django is interesting, as it seems to be the tool of choice for the best new news sites (AnnArbor.com, for instance). But, still, you gotta host it.

  9. MarkS says:

    I read this post yesterday and had to take a day to digest it. My first reaction was ‘Holy S***!’. That continues to be my reaction.

    First, thanks for sharing everything, the good and the bad. As someone with a train just building steam, it’s invaluable knowing what might be ahead.

    My response to any questions regarding my/our developing wpmu/buddypress project has always been, “Jim Groom is doing this at umwblogs”. While I acknowledge this response provided little assurance, I figured their ignorance regarding the Reverend and umwblogs only reinforced why no further explanation was needed. Jim’s feeds are spread far and wide, in fact didn’t they know that they ‘log on to the Jim Groom each morning’?

    I’ll be reconsidering the scope of my project. To be fair, the appropriate scope has always been a thought in the back of my head. I kept it in back by telling myself, ‘only an hour of support is needed a week’ and ‘Jim Groom is like King Midas’.

    Jim, again, thanks and keep sharing your experiences. It’s been a great trip. Travel long enough and there is bound to be a fender-bender or even a wreck. As long as we walk away and act on what we learn, it’s not the end.

    btw, I didn’t post our site’s address this time. I’m trying to limit the load on my server 😉

  10. Zach Davis says:

    Awww come on people – the UMWblogs beast just need a little shot of adrenaline. This problem was nothing that could be solved by dual quad core XEON harpertowns, a little bit more RAM, and a trip across dallas to a new data center. Another couple of hundred a month on hardware and a bit more storage redundancy — pennies in the world of enterprise tech spending — and the beast is humming along nicely. I’d say it’s now using about 1/10th of its capacity and all systems are go.

    Long live doin’ it on the cheap and long live UMWBlogs!

    Zach

  11. Reverend says:

    Zach,

    yeah, you’re right, it was over reaction on my part, but UMW Blogs is back up and running, just needed a little investment. I even acclaimed it today here in Puerto Rico, and I have to say that CIC rocks. But my next post, when I have time to write it, will make all this clear. The bava was down, but now it’s back up, and the mood swings just keep on coming 🙂

Leave a Reply

Your email address will not be published. Required fields are marked *