Freebase has passed a notable milestone.  On Sunday, at about 11:00am PST, we zoomed by our 10 millionth topic — and by the time you read this post, we should surpass the 11 million topic mark.  A year ago, Freebase stood at just over 4 million topics.  That’s an annual growth rate of over 100%.

A great deal went into achieving this milestone — contributions from prolific community members like tfmorris, pak21 and sprocketonline (see our contribution leaderboard for more); new Data Team tools like the recon service, RABJ, and the spreadsheet loader; and continued growth in traditional data sources like Wikipedia.  But the largest segment of growth came from our continuing efforts to build a comprehensive repository of high-quality information about media in all its forms — especially music, movies, TV and books.

In October, we rounded out our TV domain by synchronizing with the excellent user-curated TV fan site TVRage.com.  Combined with earlier data loads from thetvdb.com, we now have comprehensive coverage of nearly every TV show and episode created in the United States.  It includes cast and credits, as well as links to key TV websites like tvguide.com and Hulu — nearly a million topics in all!

But the load that took us over the 10 million mark was the final load of editions from Open Library.  Compromising 650,000 authors, almost 2 million books and 2.1 million book editions,   this load pushed new boundaries in our data acquisition, curation, reconciliation and QA processes.

In the months ahead, we’ll be continuing to both curate and extend our media data loads with more high-quality data sets.  We plan on continuing to reconcile authors and books already in Freebase, as well as loading more books from curated bibliographic catalogs.  We’ll also be fleshing out our data about movies with data from Netflix, as well as restarting our regular synchronizations with MusicBrainz and their Next Generation Schema.

Congratulations to everyone who helped get us to this point.  It’s been an exciting year — with more great data to come!

This is largely thanks to two big loads from the data team: first up, a massive import of millions of books and related information from the Open Library Project, and secondly a big load of around 255k TV episodes from TVRage.

Here’s a chart showing topic growth in Freebase since the beginning of April:

Topic growth in Freebase, April to August 2009

To take a look at the relevant data, visit our Publishing Commons or TV Commons. For some apps taking advantage of all this data, look at Tippify for book recommendations, or Alex’s TV4Me app (currently in development), a mashup of Freebase’s TV data with US broadcast schedules.

We’re big fans of the ISFDB.  Fan-created and maintained, it is widely considered one of the most authoritative sources about Science Fiction, Fantasy, and Horror literature available on the Internet.  It obsessively documents the literary careers of thousands of authors, including the series they’ve created, the titles they’ve written, and the awards they’ve won.  And, in great Open Data fashion, it’s available under a Creative Commons license.

And now the contents of the ISFDB are available in Freebase.  On our first load, we’ve focused exclusively on the book-length works within ISFDB: novels and short-story collections, nicely complementing the 100K books loaded by Toby back in January.

As always, we’ve taken great care to reconcile these new entries from ISFDB with items already in Freebase.  When we do this, we split our load into three groups: items that are clearly not in Freebase (new); items that clearly match to something already in Freebase (match); and items where we can’t make an automated decision given the available data (unclear).  We’ve already loaded items from the first two groups.  Items in the “unclear” group still need to be manually reviewed before they can be loaded.

• Authors: 9,212 new and 3,226 matched (67% coverage)
• Titles: 32,945 new and 12,900 matched (70% coverage)
• Editions: 88,697 new and 12,845 matched (87% coverage)
• Series: 5,035 new and 139 matched (96% coverage)

So you can enjoy Anathem, The Graveyard Book, Little Brother, Saturn’s Children and Zoe’s Tale, along with comprehensive views of the works of Cory Doctrow, lists of all editions of Coraline, or the publication history of Stephenson’s Baroque Cycle.

Coming soon — cover art, awards, and better reconciliation coverage.  Enjoy!

People sometimes ask use, Why haven’t you scraped infobox X from Wikipedia? or Why don’t you just load all the data from site Y? The short answer is, in many cases, we have fetched that data. And yet we haven’t loaded it into Freebase.

Why not?

Well, information we gather en masse is seldom 100% correct. For example, we might extract a list of fashion designers from Wikipedia’s category of the same name, intending to assert the fact Profession: fashion designer on the relevant Person topics. But what if some fashion houses or labels had snuck into the list of fashion designers? We don’t want to accidentally type those companies or brands as people.

In general, when our data team collects information like this, rather than making assertions immediately in Freebase’s graph, they consider it to be partial evidence and it goes into a partial evidence store. To get it out of that store and into Freebase, it either has to be

• confirmed by one or more other sources
• reviewed by a human being, or
• the original source has to be proven sufficiently accurate by statistical methods.

That’s where human intelligence and crowdsourcing comes into it. Recently, some people on our Acre team have built a series of game-like interfaces to the data team’s partial evidence store, so you can help us determine the accuracy of the assertions in there.

The first is Typewriter, which asks you to confirm guesses about the Freebase types to assign to topics we got from Wikipedia category pages.

Here you can see a topic, “El Show de las Doce”, along with a short blurb extracted from the Wikipedia article. Reading the blurb, you can tell that this is in fact a TV show. So you vote “Yes”, and the assertion is made in Freebase. Hurrah, more TV shows.

Here’s another topic we picked up from the Wikipedia category, Idol television series. If you look at that category, you’ll find that many of the topics linked, like “Canadian Idol” or “Malaysian Idol”, are television programs, but by no means all of them.

The topic shown above is actually a book written by an “Idol” judge. Using Typewriter, you would vote “No”, because it’s not a TV program.

That “no” vote makes it back to our partial evidence store and becomes a source of statistical evidence: inclusion in the “Idol television series” category on Wikipedia isn’t strong evidence that something is a TV program. (Conversely, some of the type assertions from Wikipedia categories are very strong: we’ve found that categories like 1923 deaths are extremely strong evidence that someone is a deceased person.)

But quite apart from all that heavy theory, Typewriter is just good fun. There are keyboard shortcuts, an iPhone version for using in dull meetings on long commutes, and a leaderboard showing who’s been most active player this week and over all time. You can see from the stats there that we’ve had 167,012 votes so far, of which 81% (over 135,000) were “yes”, meaning we’ve managed to make that many assertions in the graph.

Take a moment to check it out, and let us know what you think!

The first set of data from theTVDB.com — a creative-commons database for TV — has finished loading on our production site.

This gave us 150,000 new episodes and 9,000 new seasons across 3,000 series. Some examples:

• Mythbusters
• Planet Earth
• Baywatch
• Lazytown

If your favorite show doesn’t look complete, do let me know, but fear not! In the coming weeks I’m hoping to make these incremental improvements:

• improved coverage for popular shows via less conservative reconciliation
• episode screenshot images
• episode summaries
• actors and directors
• and regular syncs, to keep up with your favorite shows

All this data is, of course, available for apps and mashups via our hosted app development environment Acre, or using our REST API.

Do you recognize this man?

Believe it or not, he’s one of our biggest contributors to Freebase.  He goes by many names that you may have seen on our pages — mwcl_wikipedia_en, gns_bot,  mw_template_bot, mw_prop_bot, mergebot are just a few of his many aliases.  He works tirelessly day and night.  And, as you might have guessed, he’s not even human.

He’s Bot — the collection of all of the automated processes in Freebase’s data mining pipeline.  Freebase thrives on the content that you, our user community, creates.  But we also want to make sure that there’s a a wealth of raw information already there for you to draw upon when you’re adding your data.  That’s where Bot comes in.  By automatically gathering, cleaning and formatting data from around the web, Bot provides the Freebase community with a huge amount of data for “free”, letting you focus on the specific information your Base, View or Schema requires.

Bot starts his processing with that amazing repository of online information, Wikipedia.  Over the past 7 years, Wikipedia has seen remarkable growth.  There are currently 2.5 million articles in Wikipedia, on topics as wide-ranging as people, places, history, politics, music, culture, and science.  There are articles on companies, consumer goods, and pop media — and the number of articles is still doubling every 12-18 months.  If a topic is of any importance at all, it’s a good bet that someone has written an article about it in Wikipedia.

But Wikipedia is written by humans, for humans.  This is great if you need learn about something, or quickly look up a fact.  But there are things that are difficult to do in Wikipedia.  It’s hard to ask questions, like “What movies by George Lucas has Harrison Ford starred in?” or “Who are all of the characters in the Final Fantasy franchise?” or “Who are all the Popes of the Roman Catholic Church?” or “What are the tallest buildings in San Francisco?“.  It’s also hard to build applications on top of Wikipedia’s unstructured data.

Bot, however, has a way of extracting information from Wikipedia, known as WEX (short for “Wikipedia EXtractor”).  Run on Freebase’s Hadoop cluster, WEX is a MapReduce process that translates the Wikipedia’s markup language into an easy-to-parse XML format.  With this XML format in hand (or actuator, as the case may be), Bot can then identify the articles, redirects, and IDs of the new wikipedia pages, along with a wealth of associated information like article blurbs, templates, images, and categories.  (As a side note, Freebase makes downloads of our WEX-parsed files freely available to the community on a quarterly basis.  If you’re computationally savvy, check it out — it’s a treasure trove of raw data.  And if you need processing tools, you may want to check our Happy, our open-source Python extensions to Hadoop.)

Bot then goes through and looks for new articles in Wikipedia that weren’t there the last time he checked.  After filtering out unsuitable articles (lists and the like), he collates the article title, the redirects that point to it, a 1200-character blurb, and an image or two, and creates a stub article.  The article title becomes the name, and the redirects become aliases.  At the end of it all, Bot gives a “stub article”, just waiting for a user like you to come along and add it to a base.

But the real value comes from the semistructured data that is available within Wikipedia.  Consider the “InfoBox“, a feature often seen on Wikipedia pages.  Wikipedians use these templates to organize key information about their articles while giving a common look and feel.

If you look behind the scenes at Wikipedia, you’ll see that this handy summary is created by a bit of structured wikimarkup text, shown on the right.  Bot *loves* data like this.  Here, in nicely-structured, computer-readable text, are a variety of assertions that look very much like freebase properties.  Not only does it tell us that “Boeing 777 is an aircraft” (or, in freebase terms, “/en/boeing_777” has a type of “/aviation/aircraft_model“, but it also tells us many of the properties of that type.  We also know that “Boeing 777″ has a “manufacturer” of “Boeing Commericial Airplanes” (or, in freebase-speak, /en/boeing_777 –> /aviation/aircraft_model/manufacturer –> /en/boeing_commercial_airplanes), and that “Boeing 777″ had a “maiden flight” on “Jun 12, 1994″ (/en/boeing_777 –> /aviation/aircraft_model/maiden_flight –> 1994-06-12).

Since WEX parses all of the InfoBoxes properties (and other templates) within wikipedia — all 45 million of them! — Bot has them at his disposal to create facts in Freebase.  Of course, he needs humans to tell him what the right property is for a given InfoBox — but we’ve fed him thousands of these mappings already, which has let him assert 3.9 million high-quality properties based on InfoBoxes.

Similarly, Wikipedia category pages contain information about the types of the topics listed in the category.  For instance, if you see “San Francisco Giants” on a category page called “Baseball teams in California“, you can be reasonably sure that “/en/san_francisco_giants” should be typed as a “/baseball/baseball_team“.  WEX has over 7 million category-article associations like this — from which Bot has asserted over 1.2M types.

This is just the start.  Here at Freebase, we’re constantly adding new functionality to Bot.  For instance, there are Bots to make sure that all valid mailing addresses in the US are geocoded with a valid latitude and longitude, so that they can be visualized on tools like Google Maps.  We also have Bots to load data about newly released films, including actor and director information.  There are Bots to clean up messy and invalid data, Bots to process scheduled merges and deletes, and Bots to find new images for topics.  The are even bots that mark the passing of the dearly departed. And more is planned — soon we plan on having Bots that provide you with data about books, films, consumer electronics, and more!

Do you have tasks that you think Bot should be assigned?  Data that he should focus his unwavering attention on?  We’d love to hear from you — let us know your thoughts!

The subject data combined with the cover art makes for very cool galleries using the new topic views, here are a some examples:

• Books about animals
• Books by Warren Wiersbe
• Cookbooks
• Romance novels about Vampires

For many of the books, we also have edition data, which includes things like when the edition was published, its ISBN and Library of Congress codes, and the number of pages. This lets you sort the editions by how long they are — here are the longest editions in Freebase.

We’d love to see what collections you can come up with. Leave a comment if you find a set of books that’s particularly striking.

• 1,200 saints
• 1,900 new airports
• 2,700 beers from the Oxford bottled beer database
• 21,310 images from non-English-language Wikipedias
• 31,212 quotations from QuotationsBook
• 81,000 US place names from the GNIS database

Mmmm, tasty tasty data!

I hadn’t heard about any big new data loads, so I went looking for what had caused this sudden spike in our figures. Well, it turns out we’d had that data all along! There were a number of topics in the system that weren’t typed as /common/topic, so weren’t showing up in the count. For example, a bot which had added a number of people (type /people/person) had omitted to co-type as /common/topic, and that doesn’t happen automatically when a bot does it, only when done via the UI>.

Justin discovered this the other day, and since it’s obvious that people are also topics, and so on, he fixed it. So that’s how we’ve got so many “new” topics.

Since you can’t see the slides very well because of the lighting in the video, here is his Powerpoint deck uploaded to slideshare.net:

| View | Upload your own

Don’t forget there’s a Freebase User Group meeting next Tuesday, June 17th, in San Francisco. If you’re in the Bay Area you should definitely try to make it! RSVP at upcoming.org.