I Swivel Data
March 28th, 2008(This post is the second in a three-part series on data-sharing websites. Part one is here.)
I’d heard of the data-sharing site Swivel some time ago. When I first went there, there weren’t a lot of datasets uploaded, so there wasn’t much to look at. The site was in a pretty rough-and-ready state, too. So I figured I’d just check back later, and some months later, I did just that, leading to this write-up.
Swivel is a website started in December 2005 by two physics majors named Dmitry Dimov and Brian Mulloy. They call themselves “data nerds”, a term I’d never heard before going to Swivel but which I guess applies to me as well. (In fact, I secretly kind of like that term, along with the sobriquet of “data vulture” , a nickname regular readers of this blog will recognize.) Anyway, their goal was to make a “Youtube for data”, where you can share data freely and have other people view, analyze, and comment on that data. That’s a lofty but excellent goal, I must say. (You can read about their mission and get some background about them and their site here.)
So if you want to make a “Youtube for data”, uploading data should be easy, of course. And it is, more or less. You can copy and paste data, upload it from a comma separated values (CSV) file, or upload it direct from a Google or Excel spreadsheet. Also, there’s some tutorial videos to help get you started, which I thought was nice.
However, your data doesn’t always come exactly the way Swivel wants it. If your data is in anything other than a one row->one record format (like in a database), you need to clean it up before you can upload the data into Swivel. That means stuff like Excel pivot tables, and data that only lists row and column headers once (when they’re repeated), won’t work. Thankfully, Swivel has a nifty toolbar for Excel to help you out.
I downloaded the toolbar, and it’s pretty cool, I have to admit. (Yeah, it’s the data nerd in me talking, I know.) You can see it in action here. (The Quicktime tutorial off the Tutorial page seemed broken, for some reason.) There’s some shortcuts for various Excel functions, and a few cool add-ins like “super-fillers” which intelligently auto-fill row and column headers for you. The best part is the “matrix to table” macro, which basically takes pivot table-style data and turns it into one row->one record-style data like you need at Swivel. All you have to do is specify the row and column headers. Being a useful feature in its own right, it’s worth getting the toolbar for that one feature alone. (As they note in the video, it’s the most powerful tool in the toolbar.)
Finally, there’s a button for uploading your dataset directly to Swivel. When you push the button, it grabs the continuous “block” of data and opens the Swivel upload link in Internet Explorer. (Why no Firefox option? Swivel’s site looks much better in Firefox anyway, especially the graphs.)
One way or another, once you get your data cleaned up and into Swivel, you have to go through a six-step wizard to prepare the data. The wizard is mostly pretty helpful and smart, so it’s not a big deal, but you may have to wrestle a bit with data types and such to get it to work. The real problem is that editing your data (especially the column names and types) is kind of a pain in the neck right now. I honestly think it’s easier to upload a whole new dataset rather than try to fix a dataset you mess up. If you want to upload data to Swivel, try to make sure it’s exactly like how you want it before you go to their site. (In Swivel’s defense, I’m sure this will be less of an issue in the future. The site is still in beta and I see they’re adding tools to make data editing easier.)
And that’s basically it for uploading. You can also add a picture, categories, tags, and a description to represent your data, which adds a bit of a personal touch. In addition, your data source (which you must provide) is clearly visible on the page with your dataset, which is great if you have a hyperlink to it. You’re free to edit a lot of aspects of your dataset once you have it uploaded, but like I said, some parts are much easier to edit than others.
What sorts of data do people upload, then? All kinds of stuff. Just casually browsing I found datasets on American beer, average gasoline prices over time, most played board games in 2007, and the percentage of people that believe in God, by country. There’s also a constantly updated featured graphs section, a most viewed graphs section, and a highest-rated graphs section, among others. And if you read media coverage of Swivel, you can see they’ve teamed up with the Development Centre of the Organisation for Economic Co-operation and Development (OECD) and the World Health Organization (WHO) to provide their data on Swivel. (OECD data is also used by Gapminder, which I blogged about in part one of this series.) That means there’s plenty of ways to find cool data and charts. Good stuff.
Of course, since Swivel is all about collaboration and data sharing, I figured I ought to try it out for myself. (Would any self-respecting data nerd do otherwise?) I dug into the various datasets I’ve used for this blog in search of something to upload. Swivel works best with simple, tabular datasets that don’t have a lot of calculated fields and don’t need a lot of explanation, outside of some generated graphs and a description. A lot of datasets I had didn’t fit that criteria, but the one I used for my recent post on Biblical baby names worked nicely. All I had to do is pare the data down to the bare bones, clean it up a bit (I didn’t need the toolbar since the data was mostly in the right format), and upload it.
I had to play around a lot and do a fair amount of learning (and cursing) before I could get the data the way I wanted, though. This is one way that Swivel is not as user-friendly as it could be (to put it nicely). The wizard works great, but if you have to do something non-standard or change something, the data upload process breaks down somewhat. At first, I tried to fix my incorrectly-formatted dataset after the fact, but like I said, that can be frustrating. (For example, when I updated the name of my categories from “Year” to “Decade”, it wouldn’t change the name of the graphs.) I ended up deleting my dataset 3 or 4 times before getting things right. It was probably for the best, because I had my dates in the wrong format at first (as text strings) and I was using a copyrighted piece of clip art for my dataset picture. (I replaced it with a cute picture I found of a baby in a Christian nursery, which I assume was fair game. Who knows - that baby might even have been reading a Bible!)
You can see my mostly-finished result here. I’m fairly proud of how things turned out. I used a color wheel to pick out hex color values for the data series, and I customized the popular graphs to show the data in a more understandable way. The whole process probably wasn’t as easy as it could be, but like I said, Swivel’s still in beta and things look pretty great so far. I am confident they will address most of my issues in time.
At this point you might be wondering, “This is a cool site, but how do they make any money?” (Swivel, that is. I already know you know you think my site’s cool. Right? RIGHT?) I found myself wondering the same thing, in fact. If you read that background link I posted earlier, you saw that they’re coming out with another version of Swivel called Swivel Private Edition. Basically, if you post your data publicly, it’s free to use on Swivel. If you post your data privately - where other people can’t see it - it costs money.
Why would you want to do that? Well, another cool feature of Swivel is that you can take any two comparable datasets (say that both have dates by year) and do a “mashup”. If your data is private, you can protect your data while still being able to link your datasets to others and make mashups. Swivel reckons that’s worth paying for, and it seems like a sound idea as far as I can tell. Swivel Private Edition isn’t out yet, but since it appears to be the only potential source of revenue for the site (which has been around since December 2005), you can bet it will be out soon.
Doing a mashup is pretty easy. Here’s one comparing the Number of Catholic Priests in Missions vs. Total % of Kids With Biblical Names (my dataset). From 1960 on, you can see that the number of Catholic missionary priests has gone down sharply (which is also a more general trend) while the number of kids with Biblical names has risen. It’s not the world’s most mind-blowing correlation, but it just shows you how easy it is to do data mashups. You can even put together nonsense graphs like Historical Gas Prices vs. Total % of Kids With Biblical Names. (Gas prices are more volatile, obviously.)
So, is there any data on Swivel about blogs? Turns out there’s an excellent dataset on “why bloggers blog”. Looks like my major reason for blogging, “to share practical knowledge or skills with others”, is a respectable fourth. Ironically enough, “to make money” is a distant last as both a major and minor reason. Is that being idealistic or realistic?
(By the way, the title for this post is taken from the Swivel “data badge”. Being a data nerd, I’m sure I’ll put it up here when I get time.)
| | | del.icio.us |
April 3rd, 2008 at 12:44 am
Yes, Swivel is pretty cool. You might want to eyeball this post http://inquisant.com/?p=4 I wrote in November about them.
April 4th, 2008 at 12:59 am
[…] is not published freely. (There is a sad underutilization of data-sharing sites like Swivel, which I blogged about last […]
June 28th, 2008 at 9:01 pm
Hey Dave,
I video-interviewed the Swivel team this week and was wondering if I could use images from your screencast demo of the Swivel toolbar for Excel.
I’ll mention your site in the video and in the article that will illustrate it.
Cool? Let me know:
interview@hyveup.com
Xavier, that’s actually not my screencast. I’m not sure who made it (couldn’t tell by looking at the page on Youtube), but it’s not mine. Sorry I can’t be more help!
- Dave
July 1st, 2008 at 8:33 pm
ok thanks David