Freebase has passed a notable milestone.  On Sunday, at about 11:00am PST, we zoomed by our 10 millionth topic — and by the time you read this post, we should surpass the 11 million topic mark.  A year ago, Freebase stood at just over 4 million topics.  That’s an annual growth rate of over 100%.

A great deal went into achieving this milestone — contributions from prolific community members like tfmorris, pak21 and sprocketonline (see our contribution leaderboard for more); new Data Team tools like the recon service, RABJ, and the spreadsheet loader; and continued growth in traditional data sources like Wikipedia.  But the largest segment of growth came from our continuing efforts to build a comprehensive repository of high-quality information about media in all its forms — especially music, movies, TV and books.

In October, we rounded out our TV domain by synchronizing with the excellent user-curated TV fan site TVRage.com.  Combined with earlier data loads from thetvdb.com, we now have comprehensive coverage of nearly every TV show and episode created in the United States.  It includes cast and credits, as well as links to key TV websites like tvguide.com and Hulu — nearly a million topics in all!

But the load that took us over the 10 million mark was the final load of editions from Open Library.  Compromising 650,000 authors, almost 2 million books and 2.1 million book editions,   this load pushed new boundaries in our data acquisition, curation, reconciliation and QA processes.

In the months ahead, we’ll be continuing to both curate and extend our media data loads with more high-quality data sets.  We plan on continuing to reconcile authors and books already in Freebase, as well as loading more books from curated bibliographic catalogs.  We’ll also be fleshing out our data about movies with data from Netflix, as well as restarting our regular synchronizations with MusicBrainz and their Next Generation Schema.

Congratulations to everyone who helped get us to this point.  It’s been an exciting year — with more great data to come!

If anyone has some spare time today, we could do with your help with voting on delete/merge tasks. Please head over to http://www.freebase.com/tools/pipeline/showtask and contribute some clicks.

There are loads of topics waiting to be voted on. They were flagged for delete in bulk and been a bit too keen, so there is now a bit of backlog to be worked through.

Some of the topics for deletion are lists – the names of which fit the pattern “National Register of Historic Places in {location}” and “{country} films of {year}”.

And others are topics about topics e.g. “History of x”, “Economy of x”, “Politics of x”, “Communications in x” etc. – the data in these really belongs on the topic x itself, but they can be dealt with later.

Hopefully a bit of crowdsourcing will get the backlog down to usual levels, so please help out at http://www.freebase.com/tools/pipeline/showtask.

Thanks,

Iain

The Freebase data dumps are current as of October 1st, and the WEX (Wikipedia extract) dumps as of September 12th.

Philip’s email explains what you’re seeing:

* Nodes represent types, edges represent properties.
* Reverse properties are shown in parentheses after the master property name.
* Grey edges represent hidden properties
* Blue nodes are types outside the domain being viewed, but which are linked to from the domain (the edge will be dashed if the master property is from the type outside the domain to that inside the domain)
* Red nodes represent undocumented types
* Node shapes indicate the display style of the type (ellipse: standard, rectangle: CVT, triangle: enumeration)
* All nodes are clickable to view that type

The visualisations are now built into the schema explorer.

As with all other Acre apps, the source code is available and cloneable.

Brian Karlak looked into it and told me:

We have ~600K people in Freebase with both names and genders.

I took the first name as the first space-separated token in the /type/object/name of these people. Dreadfully simplistic, I know, but it will do for this analysis. It does mean that “first names” will include such honorifics as “King”, “Princess” and “Dr.”, but that’s OK — some correlate very well with gender.

I looked for first names with at least 100 exemplars. Of those, there were 693 first names that correlated >99% with a particular gender. There were 507 that showed 100% correlation with a particular gender. Over 80% (412) of these were male names.

There were 127 first names that had at least 100 exemplars but did not correlate consistently with a single gender. The most gender-ambiguous name was Andrea (nearly 50/50 split), followed by Ashley (48/52), Dana (53/47), Nicola (47/53), and Charlie (47/53).

Brian tells me we’ll be able to use this to assert the genders of people in Freebase, so you don’t have to.

If you’re interested in gendered names in Freebase, I recently created an app to do more or less what Brian is doing above. It’s called Gendered Names and if you tell it a given name, it will tell you the split between male and female based on what’s in Freebase and present it as a nice chart. Here’s what it thinks about the name “Evelyn” for example:

Of course, as with any Acre app, you can view the source and clone it if you’d like to build something similar.

This is largely thanks to two big loads from the data team: first up, a massive import of millions of books and related information from the Open Library Project, and secondly a big load of around 255k TV episodes from TVRage.

Here’s a chart showing topic growth in Freebase since the beginning of April:

Topic growth in Freebase, April to August 2009

To take a look at the relevant data, visit our Publishing Commons or TV Commons. For some apps taking advantage of all this data, look at Tippify for book recommendations, or Alex’s TV4Me app (currently in development), a mashup of Freebase’s TV data with US broadcast schedules.

And now — with your help — Freebase can, too!

http://detypewriter.freebaseapps.com/

Written by vtalwar, (de)Typewriter is a Typewriter-style application based upon our incompatible types definitions.  On a nightly basis, we check our topics against these rules.  Topics that have incompatible types are marked and flagged for review and cleanup.

That’s where you come in.  Cleanup of topics with incompatible types isn’t necessarily straightforward.  In some cases, the incompatibility is just caused by a simple mistyping — a book was mistakenly typed as a person, or a band was mistakenly typed as an album.  These cases are easy to fix: the offending type just needs to be removed.  There are other cases, however, which are more difficult to fix — for instance, in cases in which two entities were mistakenly merged. This requires that the offending topic be split into multiple topics, which is a much more involved process.

For instance, the band “Happy Mondays” is not and never will be a /people/person, so it needs to be detyped.  However, “Bob Dylan” can be both a /people/person, and his eponymous /music/album, so it would need to be split.(*)

Once we know which case we’re dealing with, we can make the fix.  But we need your to help us figure out which case we’re looking at.   Much like the Typewriter and Genderizer apps on which it’s based, the (de)Typewriter application walks you through a list of potential incompatibilities.  But in this case, our question is phrased differently:

First off, you’ll notice that we’re not asking about a particular Freebase topic.  That’s because we already know that the Los del Rio topic may have problems.  Instead, we want to know if somewhere, out there in the big wide world, if there is a person named Los del Rio.  If there is, then we know we have some split-like cleanup to do.  If there’s isn’t, however, then we know we can automatically remove the /people/person type from the Los del Rio topic.

For each question, you’ll get some clues.  On the left panel, you’ll see the results of a wikipedia search for that name.  On the right, you’ll have Google, IMDB, or Google Maps results.  Usually the answer will be obvious from these clues — but feel free to follow links if you think more information will help.

Try not to “overthink” the questions.  If the answer isn’t obvious (or if a question and/or its supporting clues seem confusing to you), just click “Skip” to go to the next question.  You can also get extra help by clicking the “Help” button, which will give you a discussion of the type’s usage as well as letting you know which incompatible type rule spawned this question.

Finally, note that unlike Typewriter and Genderizer, (de)Typewriter uses a vote tallying mechanism.  Detyping doesn’t occur immediately after clicking a button.  Instead, votes on the separate typings are compared to determine the proper course of action.

Have fun detyping!

If you’re like me, MQL often feels like a combination of power and dark magic: some people can whip up a MQL query in no time, like they knew all about every type, every property and how they link all the various pieces of the graph together.

Asking around how people did it, yielded several strategies… but many basically revolved around using the ‘explore pages’ (those magically revealed by hitting the F8 key on freebase.com).

Then David wrote the new query editor, with the amazing context completion and I thought I was set… but while my MQL mojo improved a lot with it, it always felt like something was missing: if I knew the type I wanted, I could get started… but what if I didn’t know where to start? what if I didn’t know how to construct the query and I needed some guidance in navigating how the types were all connected together? (especially with CVTs and whatnot)

I felt like I was in a fog and with no map: finding my way out felt tentative at best, frustrating at worst.

The Jason showed me an acre app he had written a long time ago and that he uses all the time that offered a browsable interface to all the freebase domains, types and properties, their IDs… and how they all connect together.

I had found my map.

Unfortunately, while extremely useful already, the schema explorer app needed some love… so I decided to give it some and the result has been so successful internally that I’m happy to announce it to the public so that everybody can benefit from it. Find it at: http://schemas.freebaseapps.com/

I hope you’ll find it as useful as I do.

NOTES:

1) suggestions/criticism/feature-requests appreciated and welcome

2) this is an acre app and if you’re curious (or interested in forking it) you can find it at: http://acre.freebase.com/#app=/user/jdouglas/schemas

3) this couldn’t have been possible without Jason’s work upon which I built this so kudos go to him while all the mistakes remain mine

4) I’ve been working on integrating the schema explorer with the upcoming freebase developer’s documentation hub, so that newcomers won’t have to hunt down this URL in the mail archives to find a little guidance in the type maze.

Enjoy!

Reconciliation gets better the more we know about the topics in Freebase, and one of the most important things to know is a topic’s types. That is, when we’re trying to match an author “Jane Smith”, it helps if we can limit our matching to people named Jane Smith that we already know are authors.

In short, typed topics make our data loads more effective.

We’re already working on typing 95% of the topics in Freebase. If you’d like to help out, and know that your efforts are going to be multiplied by the data team’s mass loads, you can help us by typing authors and books.

Using Typewriter, simply choose Author (15,349 items) or Book (9,323 items) from the list of types, and answer “Yes” or “No” for each topic.

Don’t forget, there’s also an iPhone app: great for that long commute or boring meeting!

For more background reading on our data loads, reconciliation, and human intelligence, check out:

• Meet the Freebase bots
• Partial evidence and human intelligence: a dispatch from the data mines