Cатсн²² (in)sесuяitу / ChrisJohnRiley

Because we're damned if we do, and we're damned if we don't!

Tag Archives: rss

{Quick Post} Adding rand() to Yahoo Pipes

This is just a quick post (and not particularly security orientated) about some modifications I made to the patched together Tumblr –> Yahoo Pipes –> FeedBurner solution I use for the Suggested Reading links I post on twitter (and make available through feed.c22.cc).

The problem:

I’ve begun to notice that any blogpost or articles I share recently are turning up on Twitter (the ones with the dreaded [SuggestedReading] tag) about 6 hours after I actually share them. This doesn’t make for a very reliable sharing system as people in InfoSec tend to want up-to-date information, not day old data clogging their time lines.

The thought process:

Originally I thought this was a problem with feedburner as it was the last item in the chain and I know it doesn’t update realtime. The feedburner section of the process adds items to the twitter stream with the prefix [SuggestedReading]. As it turns out this didn’t seem to be the cause of the problems, as even a forced refresh wasn’t doing the trick. Following it back through to Yahoo Pipes I saw that the feed.c22.cc/rss feed was loading, but always responding with what seems to be a cached version.

The fix:

The fix I implemented was to add a ever changing string to the end of the Yahoo Pipes “fetch feed” to prevent the caching issue.

Instead of requesting feed.c22.cc/rss it would request feed.c22.cc/rss?timestamp=1338531580

As Yahoo Pipes doesn’t offer a rand() function, I implemented a series of pipes that take the current date/time, transform it into a string (e.g. 1338541580) and then add it to the end of the static URL using the Yahoo Pipes URL builder function.

You can view the Yahoo pipe here

Rewriting Tumblr RSS feeds

After the demise of Google Readers sharing function (thanks for that Google), a lot of people in (and out) of the InfoSec community searched about for a suitable replacement, without much joy. As a stop-gap solution I moved over to Tumblr (feed.c22.cc) and starting using it to share interesting things through an RSS feed and have that reposted to Twitter (see my [SuggestedReading] tweets). This seemed to work as well as any solution, but there was one nagging issue that kept bugging me. When you clicked on a [SuggestReading] link posted to Twitter you were redirected to Tumblr, and given the real link to click to see the story. A small issue, but something that bugged me, and bugged people using those links as well…

Feed from Tumblr (example entry):

<item>
<title>
Oatmeal: I tried to watch Game of Thrones and this is what happened
</title>
<description>
<a href="http://theoatmeal.com/comics/game_of_thrones">Oatmeal: I tried to watch Game of Thrones and this is what happened</a>: <p>… and THIS is why people pirate shit!</p>
</description>
<link>http://feed.c22.cc/post/18002558690</link>
<guid>http://feed.c22.cc/post/18002558690</guid>
<pubDate>Tue, 21 Feb 2012 03:19:37 -0500</pubDate>
</item>

As you can see the link tags point to posts on feed.c22.cc (the tumblr blog)… and not direct to the end URL.

So, in a moment of frustration I sat down and wrote some PHP code to rewrite the RSS feed. It’s not well written, and it’s not perfect (infact I struggled a bit with some UTF-8 encoding issues, which I HOPE are now fixed). In the spirit of sharing, I’ve uploaded the source incase anybody with the same issue wants to host their own script to perform rewriting.

Rewritten feed (example entry):

<item>
<title>
Oatmeal: I tried to watch Game of Thrones and this is what happened
</title>
<description>
<a href="http://theoatmeal.com/comics/game_of_thrones">Oatmeal: I tried to watch Game of Thrones and this is what happened</a>: <p>… and THIS is why people pirate shit!</p>
</description>
<link>http://theoatmeal.com/comics/game_of_thrones</link>
<guid>http://theoatmeal.com/comics/game_of_thrones</guid>
<pubDate>Tue, 21 Feb 2012 03:19:37 -0500</pubDate>
</item>

The rewriting process is called each time the PHP file is requested, but this can easily be scheduled and output to a file if you need.

Process:

  • Feedburner calls rssRwrite.php (self hosted)
  • rssRwrite reads in the Tumblr RSS
  • Entries are extracted from this RSS
  • A new RSS is created (with required changes to the link)
  • This new (rewritten) RSS is returned to Feedburner
  • Feedburner does it’s thing!

Personally I setup feedburner to access the rewrite PHP link and republish (and share out) the content as required. This step is up to you, but to reduce load on the rewrite script this seemed like the best trade-off, and I use feedburner for sharing things anyway. It’s a bit of a tangled web, but one that seems to work for now!

Hope you enjoy… and please, no laughing at my bad PHP code ;) comments are, as always, welcomed!

Update:

@mubix pointed me to Yahoo pipes as an easier alternative to achieve the same kind of rewrite… You can cehckout the solution he suggested HERE. I hadn’t really looked much at Yahoo pipes, and TBH, thought it has been discontinued as the Yahoo empire began sinking into the sand from whence it came. Good to see it’s not only still available, but actually one of the few Yahoo resources that is actually useful ;)

Links:

  • rssRwrite PHP source –> HERE

Filling your RSS reader

As Google is making some changes to Google Reader (I say changes, but I really mean, killing it off by removing the feature we all want and use), I’m not sure what’s going to happen to the shared items feed I provide at the moment (through the blog link above and through Twitter). GReader made sharing the information from other blogs and sites really easy, and as such I’m looking for a good replacement right now… maybe using Tumblr as a replacement… not perfect, but better than a crippled GReader can offer right now.

Still, for those who’ve asked for a list of good blogs to follow I’ve exported my blog list to OPML format for you to download and import or take a look at. It’s not small, so make sure you’re ready for it if you import it into your reader software.

Google Reader Export ==> (OPML)

Temporary Tumblr feed ==> http://feed.c22.cc (RSS)

Follow

Get every new post delivered to your Inbox.

Join 129 other followers