Why we can't have nice hyperlocal things

I've been experimenting with the hyperlocal newsletter I mentioned. The results so far aren't promising.

Here's a screenshot of the latest version:

Screenshot of the current newsletter

(It’s a bit worrying that the Small Print takes up so much of the newsletter…)

Some stuff I'd like to put on the newsletter just isn't online. Local events like schools or churches having jumble sales or something. They're handled by leaflets to parents or the congregation, but they're rarely posted online.

Other stuff I'd like to put in is available online, but not in a format that can be automatically processed. (This normally means XML or JSON - I'm not going to do any screen-scraping because if they haven't deliberately chosen to syndicate their content I'm not going to use it.) Things that aren't available in a syndication format include Carrickfergus Borough Council's blog/news feed (seriously) or the several sources for "What's On" events.

Some stuff is sortof-online, like Carrickfergus Times' feed of events. Some events are indeed here, in XML, but there's no easy way to derive the date of the event. Sometimes it's there in the title as 'May 10', other times it's explained to be 'May Day Bank Holiday', other times it's not mentioned at all.

One feed that actually got the date concept right was the Parades Commission feed. I'm not even going to try and explain the reason we have a Parades Commission in the first place, but the feed used a separate <dc:date> tag to show the date of the event. Nice! Except the feed is now just returning a 404 after some site changes. So the one nice, useful event feed has disappeared.

Other stuff that's sortof-online is local planning applications that are up for approval or whatever happens with such things. They're online, but there's no fixed URL for them, and anyway they're only available as PDFs...

Then there's the stuff that is available online, and is in a suitable syndication format, but some utter fuckwit intervenes and makes it impossible to process. I'm still annoyed at the way the police make crime data available - there's no actual link you can go to to download the latest data, or download any particular data set. You have to go to a web page and click to generate the data set which then gives you a new, unique URL for the data set. You can't do this automatically, it all needs to be done through a web browser.

Then there's the bad XML which makes the whole file un-parseable. I may take a look at the parser I'm calling to see if I can make it more tolerant, but is XML-encoding < and > so hard?

Finally there's the data that's online, in a syndication format that I'm not allowed to use. I'm doing this as part of my company, so this is (potentially) a commercial venture. Some sites (like the Met Office) say their feeds are for personal use only. Some (like Google News) say if you want to use them for commercial purposes you should contact them - but then neglect to provide any actual way of contacting them!

Only a few things have been straightforward. OpenWeatherMap was nice to call, and available to use, but the weather forecasts for Carrickfergus just didn't seem to match the actual weather at all, so I'm trying Weather Underground and it seems nice (they do mandate the look of the credit notice, which may start to get ugly quickly if a few more start mandating different looks).

Tags: Development
Created by on Logo15659OpinionatedGeek Ltd.Logo15659