2010-08-15 00:04:02API feed
John Cook

john@skepticalscience...
121.222.17.49

One thing I've been asked about by a few different people is providing an API or an XML feed that other applications can use. Eg - to make the info in all the skeptic rebuttals available for other applications. I'm very keen to do this. One exciting application by Nigel Leck scans Twitter to find tweets about common skeptic arguments then automatically tweets rebuttals featuring links to Skeptical Science arguments. It would be good for the rebuttals to be able to be dropped into other websites. And who knows, there are probably other creative ways to use the info that I couldn't possibly imagine.

I've already created XML feeds for the iPhone and Android app - plus another separate feed for the Nokia app. But the iPhone feed was created specifically for the needs of the guys programming the iPhone app using specific markup tags they requested. A general XML feed needs to use generic XML structure and there's probably other technical requirements I'm not aware of yet.

One other spanner in the works is the fact that the whole system is being overhauled into a 3-level rebuttal system: Basic/Intermediate/Advanced. So those changes will need to be put into place before an API feed can be created. In the meantime, I'm open to advice and suggestions on how best to approach this whole API idea as frankly, it's something I have no experience in.

2010-08-16 11:52:33
James Frank

james.e.frank@gmail...
71.191.120.49
This sounds like an area where consolidation would be in order. There's no point in generating XML feeds off of the database multiple times, as we're just duplicating work. We need to come up with an API that allows you to query for an argument(s) and specify the return format.

For example, you might do api.php?arg=22&format=iphone to get back one specific argument for the iPhone app. Or api.php?term=solar%20flares to get back a list.

api.php would return, by default, a standard XML response we would define. If you included the format= option, it would run that through an XSLT file to give you the correct type.

It might be helpful if you posted an example of the XML you are already generating so we can see what all we need to supply.

At a minimum I would think we would need options for article id or search terms (mutually exclusive), format, and level.
2010-08-18 15:29:47
Nigel Leck
Nigel Leck
nigel@stsoftware.com...
203.206.176.223

I've been able to read all the arguments of the arguments list page ( with twitter links).

The real trick is to be able to make answers to new arguments come out quickly like this one that is currently doing the rounds :-

We would need the answer... generally easy enough to find when you go looking for it plus a matching pattern which is something like :-

  • AGW *.*  NOAA satellites *.* bad *.* data
I've got a database of all the tweets about climate for the last N months that I can test the matching criteria against so make sure that I don't get fail alarms. Also have a list of known good sources which automatically add phases to be excluded when the system finds a match but from a known good source.
2010-08-18 17:38:17Twitter app
John Cook

john@skepticalscience...
118.208.41.102
Nigel, what do you think of starting a new thread solely about your Twitter app? It took me about half a dozen emails from you before I started to get my head around what you're doing on Twitter any how it all works. I have a vague, fuzzy vision that we could somehow connect this fantastic community of climate writers and your Twitter system but I don't know how just yet. So perhaps if you could introduce your Twitter system in a new thread, we could discuss ways to have authors contribute to it so that new skeptic arguments propagating on Twitter can be quickly responded to.
2010-08-19 21:23:26Thoughts
christianhunt

christian@carbonbrief...
88.211.15.182

On the API for arguments - I think you're right that the 3-tier thing would need to be sorted first, and selectable in the API. (is that what you meant by 'level', James?

Being able to summon arguments into places on the web would be very useful. It would be useful to be able to embed arguments into web pages so that they remained up-to-date. Equivalent to the youtube embed feature...

Having a standardised, published API would make building these kinds of tools a lot easier.

Are the arguments themselves available as an RSS feed, that updates when they do?

2010-08-19 23:08:56
James Frank

james.e.frank@gmail...
71.191.120.49

Yes, that's exactly what I meant by level. John, if you'll post an example of the iPhone / Nokia XML (so I can know what all people are looking for) I'll take a stab at creating some sample documentation. We can revise that so it provides what we want, and then start building the PHP backend to fulfil those requests. How's that sound?

(Also, is there any way to get the forum to email us when someone updates a thread we're on? I'm two months out from my wedding and, to be honest, so busy that I forget to come back and check often. Having an email ping would be great.)

2010-08-20 14:53:10Examples of XML
John Cook

john@skepticalscience...
121.222.17.49

Here are the XML feeds that currently exist. They're all data-driven so they update instantly when I update the database content. I will note that these were created according to specific requests from the iPhone and Nokia developers so they aren't meant to be general, consolidated feeds. The iPhone XML in particular is quite idiosyncratic. How it works is it firsts lists the Argument Number and Update Date for each argument as a sort of reference XML:

http://www.skepticalscience.com/arguments.xml

Then each individual skeptic argument has it's own feed:

http://www.skepticalscience.com/arguments/1.xml

As for the Nokia XML feed, this isn't particularly efficient, it lists ALL the skeptic argument rebuttals in a single huge feed:

http://www.skepticalscience.com/args.xml

I'm open to advice and/or documentation on the proper way to do this that would hopefully be of use to most or any situations. I like the idea of embeddable code a la YouTube.

Good idea re email notification - I'll get onto that!

2010-08-22 13:35:26
James Frank

james.e.frank@gmail...
71.191.120.49

I’m going to make the assumption that this all operates off of a single URL (e.g. /api.php). That makes the most sense to me, for simplicity of use and ease of reference. That doesn’t mean it has to be a single file in the backend; on the contrary, it would be built from a set of PHP and XSLT files. You would determine which files were called and what the resulting XML looked like via the query parameters.

Parameters

1. Version (?v=), Required

This would take a number as a parameter refering to the version of the API we’re calling. Looking toward the future, and all that. Each version could be a separate include that would be called by api.php.

2. Query Type (?type=), Required

This would take one of several options, determining the type of query that we’re asking for. I would imagine:

  • All (?type=all): Lists all articles
  • Article (?type=article): Lists one particular article
  • Search (?type=search): Allows a search for a keyword or keywords

Examples of the resultant XML for each are provided below.

3. Format (?format=), Optional

This would take on of several options, determining the type of XML returned.

  • XML (?format=xml): Default option, returns the default XML response
  • JSON (?format=json): Returns the same data as JSON
  • iPhone (?format=iphone): Returns the XML for the iPhone
  • Nokia (?format=nokia): Returns the XML for Nokia

All formats other than the default XML would be generated by running the XML through an XSLT stylesheet. This makes it easier to rewrite the core for database changes, etc.

4. Article ID (?id=), Required if type=article

For article queries, the ID would be required and would return the XML for the article in question.

5. Keyword (?q=), Required if type=search

For search queries, this would pass the keyword(s) to the database query. Keywords coul dbe passed with + in between (e.g. ?q=water+vapor).

6. Amount of Detail (?amount=), Optional

This would take on of several options to determine how much data to return. If someone is just populating a list, they don’t need the full data for each article.

  • Summary (?amount=summary): Default option, returns id, title, URL, creation time, modification time, and the skeptic argument
  • Full (?amount=full): Provides all available feeds

Note that this wouldn’t affect the individual article type.

7. Level (?level=), Optional and only valid if amount=full or type=article

This would take one of several options to determine what levels of arguments should be returned. This brings the API in line with the recent changes.

  • Intermediate (?level=intermediate): Default option, returns only the intermediate level
  • Basic (?level=basic): Returns only the basic level
  • Advanced (?level=advanced): Returns only the advanced level
  • All (?level=all): Returns all levels

 

Those are the parameters I would imagine us needing right off the bat. Here is some sample XML for the different types. 

... Having a bad time with this editor and sample XML. See it here. ...


Ideally this would also have some sort of a caching system behind it to cut down on the database queries for the full list.

This could easily be turned into drop-in code for other websites. For example, you could create a simple page that listed the most recent version of an article’s argument and the basic response. It would do this via an AJAX call back to the Skeptical Science website. Then this page could be dropped in via iframe onto any website. You could even make a simple “generator” where you could choose what article you wanted and it would give you the code to include.

Some ideas for current or future improvements to the API might be to include really short (1 paragraph) responses, tweet responses, and comments.

 

 

2010-09-02 22:10:59Heading Out of Town
James Frank

james.e.frank@gmail...
71.178.154.171

Any progress with this? I'd be happy to take a stab at coding the PHP to return things. I am headed out of town for two weeks and then I'll be in the last month of wedding preparations, so I probably can't get a lot done before late October/November. But if it's still available then, I'd be happy to try.

2010-09-13 21:13:51Delay in progress
John Cook

john@skepticalscience...
121.222.93.62

James, deepest apologies for the lack of response. I am very, very keen to get this going, I think there's enormous potential in an API. The documentation you've done is amazing and comprehensive, in fact, a little intimidating! :-) Which is partly the reason I haven't got to it, such a big job to tackle (I'm a terrible procrastinator on big, hard tasks). I will aim to work on this ASAP but if I still haven't been able to get anything done by the time you're available to work on it, then would very much welcome your help with the coding.

Best of luck with your wedding preparations... and the whole marriage thing! :-)

2010-09-24 21:02:00Embeddable
kdkd
Kieren Diment
diment@gmail...
124.171.3.155
One of the advantages of RSS is that using standard tools it's basically pretty easy to embed entries. I'll have to look at the spec to see if it provides the keyword search facility that would be desirable.