Playing with the Data Transfer Project

I wanted to get one more post out about another project I have been playing with these days. While attending the MyData conference in 2018 I saw a presentation from folks at Google about the Data Transfer Project (DTP). The project is an effort to create data standards across major social media companies like Facebook, Google, Twitter, Instagram, Flickr, etc. that would allow you to migrate your content on one platform to another.

So, such a tool could be used to migrate all my posts in Instagram over to Flickr, which would allow me to keep the image, descriptions, likes, etc. It is a tool that could be really useful for archiving, and I immediately started likening it to the Reclaim Your Domain discussions we had with Audrey Watters and Kin Lane back in 2013. The idea was not only about moving from one service to another via API, as this project does, but also taking that same approach and using it to pull your content into an open source, self-hosted application such as WordPress. I would love to one day have a version of this kind of thing in someone’s Reclaim Hosting dashboard that provides them the ability to archive and/or migrate from a service like Flickr or Instagram into a an open source content management system or straight-up HTML. This kind of portability should be part and parcel of the move towards data protection, the ability to port your data seamlessly to avoid platform lock-in.

Anyway, while I have thought about the implications of this project, I had not played with it since the workshop the DTP folks put on in 2018 at MyData (which was quite good). So, having some time last week I decided to try and fire it back up. It was a good reason to fire up Docker Desktop client for Mac, and I followed the directions on the DTP Github page to get the demo running locally on my computer. Initially I ran into an issue that Nginx was not loading, and after asking Tim for a second opinion he confirmed as much so I submitted query to their Google Groups forum. I got a quick response, and they resolved the issue and commited the fix, so when I did a pull request from Github the fix was working. Disco!

So, above is what the demo interface looks like, and you can choose what data you want to be moved from where, the key here is that you need the Environment secrets for the accounts you want to access (so a secrets to access your data fromInstagram, Flickr, Google, etc). Once you have those environment secrets you can run the DTP demo and play with it. I was not able to get access to my Instagram or Google data (so I may need to check my secrets), but I was able to transfer images from Twitter to Flickr. Which I thought was awesome.

So this tweet…

Was pulled into my Flickr account:

Copy of - Twitter Photo 1253618782981029888

The titled comes in as “Copy of – Twitter Photo 1253618782981029888” because Tweets don’t have titles, but the text did come in as the image description, which is nice. One thing I did notice is that in the conversation GIFs and videos were simply converted to JPGs, so that could be an issue. The other thing was that only 17 images came over from Flickr, and I probably have thousands, so I am wondering if one can lift those limits, I’ll need to dig in on that.

The other question for me is can I get Instagram working. The whole reason I finally decided to do this is that after 1000 posts on Instagram I feel like I am done. I did my time, and I enjoyed it. But my job as a Instagram influencer didn’t pan out 🙂 What’s more, I really want to avoid pushing too much content into a service I can’t get my stuff out of cleanly. I’ll explore other ways to migrate and/or backup Instagram (anyone playing with this?), but until then I am limiting my posting on Instagram until I have a viable escape plan. I already did that with my Tumblr, and I am glad I did—as much as I miss Tumblr.

So, this is another work in process, but one I really would like to see flower because one of the crucial elements of the effectiveness and usefulness of legislation like the General Data Protection Regulation is that it encourages companies to make our data more portable with incentives to create data models that adhere to standards.

This entry was posted in reclaim and tagged , , , . Bookmark the permalink.

6 Responses to Playing with the Data Transfer Project

  1. Tony Hirst says:

    Have you seen Simon Willison’s Dogsheep project?

    • Jim Groom says:

      That’s a good find, I did not know about this, I am going to have to follow that more closely, thanks for the heads up, Tony!

  2. Ted Curran says:

    Hey Jim,
    I have been using Intagrate Lite to slurp my Instagram posts into my WordPress blog’s storage so I have backups of all the images. I’m not sure if it works retroactively or if it only catches new posts – actually I need to figure this out because there connection was down for a while – but maybe it’ll point the way forward.
    All the best,

    • Jim Groom says:

      Hi Ted,

      Thanks for the comment. A lot of the plugins I have seen do not work retrpactively. It will basically limit the feed to 15-20 at most. I am wondering if RSS Hub might be one way at this:

      Gonna have to play with this

  3. Alan Levine says:

    That’s rather slick that it works! Instagram is a pita since they change their API so often. I wonder if it’s worth it, as don’t you get a crappy sized photo back from IG? Do they even store originals?

    I do think it’s better to have a strategy of organizing photos locally before putting them in the cloud rather than trying to get them back out. But maybe I’m just a grouchy old man.

    I’d bet too you can hit API limits. I know most anything I’ve done even on Flickr’s own site poops out with over 60,000 photos.

    • Reverend says:

      Yeah, luckily I have the photos, but what I would like to grab is the brief description, any comments, who liked it. None of it do or die, but part of the context I would love to capture if I could. I should have definitely done something before 1000 photos, so that is on me. When will I learn?

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.