1. how to start with a reasonable CSS stylesheet that doesn’t look ugly but it’s also not a pain to use
2. a series of highly reusable HTML fragments that do the things we do all the time (like, add jquery to this page, or add a header and a footer, etc)
3. document a series of javascript libraries that enhance the default Acre environment and simplify your life

So, look here:

• http://default.freebaseapps.com/ – contains some templates that create simple HTML fragments for you to use and a default CSS stylesheet that makes your page look reasonable from the get go (the app uses its own default_stylesheet so it will look like that one)
• http://libraries.freebaseapps.com/ – lists useful JS libraries that you can use alongside with example code you can cut/paste to get you going
• http://wiki.freebase.com/wiki/Acre_Recipes – contains useful recipes and code fragments that you can cut/paste in your app to do useful things in acre that are not obvious. Now wikified! so you can add/request your own.

Enjoy!

Some other interesting tools that are part of the acre-development app include:

• Most-cloned files
• Published apps without a description (are any of yours on there?)
• An RSS feed for the newest apps by creation date.

As usual, the source code for the app is there for you to read or clone.

Quick notes:

Yoz started the show with a 15 minute version of a longer “Folk Programming” talk he does, followed by me talking about how the concepts apply to open source software, and then Jason talking about how we (i.e. Freebase.com) applied the concepts to our new app development platform to help encourage adoption and a strong developer ecosystem.

Yoz starts out talking about he first learnt to program (on a ZX Spectrum), and how beginning programmers often cargo cult as a way of learning. Cargo cult programming is often stigmatised, but it can be seen as a kind of folk process, like folk music or folk stories, and it’s a good way to get started.

He then gave a bunch of examples of folk programming in action, from an MIT study of children passing around little games on Dreamcast modules, to Yahoo Pipes and Second Life. He identifies five things necessary for a real folk programming environment:

• Create your own code.
• View, clone, and modify someone else’s code.
• Free, always-on, ubiquitous hosting.
• Huge array of diverse data sources.
• Code creates new data for use by others.

I then took over and slightly reframed his list for the open source world:

• Free and Open Source licenses
• Easy cloning forking
• Hosted development environments
• Huge array of data

I looked at some examples including github, Amazon EC2, and Google App Engine, and looked at them to see whether they provided folk programming environments for open source development. What I found is that each provides some elements, but none provides all of them.

When Jason and his team were starting work on our hosted app development platform, Acre, we got Yoz in to give a lunchtime talk about Folk Programming, and used a lot of his ideas in developing our platform. Jason walks us through how Acre has BSD licensing, one-click cloning, and interacts with Freebase’s huge store of open data.

Some of the demos he shows include “data games” for Freebase contributors to more quickly add data to Freebase itself, which started off with a single app called Typewriter and quickly evolved into a cluster of apps and eventually a library and toolkit to allow people to quickly build more. (Some were built at our recent Hack Day, for example, including one to easily find CC-licensed images to attach to Freebase topics.)

Another set of apps he showed were clustered around the concept of lists in Freebase, starting with an app to create lists of anything and then showing a recommendation website and an app for TV fans all based on the same underlying data.

Jason closed by talking about some of the challenges we’d faced and future directions for folk computing in the open source world. Challenges included version control and collaboration (because there is a time when you’re no longer experimenting and need to work together on things rather than fork), and the state of in-browser development tools like syntax-highlighting code editors and Firebug-style console debugging for server-side apps.

Questions from the audience included “Have you seen people writing large apps in folk programming environments, in the thousands or tens of thousands of lines range?” to which the answer is “yes” — LambdaMOO and SecondLife being enormous, and some of the Freebase apps being in that linecount range already despite only having properly launched about two weeks ago.

“How do you handle security?” was the second question, which Jason answered at length. Some of the parts of the Acre security model included sandboxing on the server; having each app on its own subdomain to prevent cookie hacks; and everything being sanitised by default to protect against cross-site scripting attacks.

If you haven’t looked at Acre or freebaseapps.com in a while, a lot has changed, and it’s worth a visit.

What’s new:

• The Acre platform itself is officially at version 1.0, which means API stability.
• Our new App Directory makes it easy to find Acre and non-Acre Freebase apps. You can search by app name or author, and our featured apps include some of the best ones from the Freebase developer community.
• Our developer documentation has been completely revamped.
• Our developer tools, including the query editor, app editor, and schema explorer, have been improving by leaps and bounds over the last few months.

Take a look at our screencast, too, to see what’s new:

If you’re like me, MQL often feels like a combination of power and dark magic: some people can whip up a MQL query in no time, like they knew all about every type, every property and how they link all the various pieces of the graph together.

Asking around how people did it, yielded several strategies… but many basically revolved around using the ‘explore pages’ (those magically revealed by hitting the F8 key on freebase.com).

Then David wrote the new query editor, with the amazing context completion and I thought I was set… but while my MQL mojo improved a lot with it, it always felt like something was missing: if I knew the type I wanted, I could get started… but what if I didn’t know where to start? what if I didn’t know how to construct the query and I needed some guidance in navigating how the types were all connected together? (especially with CVTs and whatnot)

I felt like I was in a fog and with no map: finding my way out felt tentative at best, frustrating at worst.

The Jason showed me an acre app he had written a long time ago and that he uses all the time that offered a browsable interface to all the freebase domains, types and properties, their IDs… and how they all connect together.

I had found my map.

Unfortunately, while extremely useful already, the schema explorer app needed some love… so I decided to give it some and the result has been so successful internally that I’m happy to announce it to the public so that everybody can benefit from it. Find it at: http://schemas.freebaseapps.com/

I hope you’ll find it as useful as I do.

NOTES:

1) suggestions/criticism/feature-requests appreciated and welcome

2) this is an acre app and if you’re curious (or interested in forking it) you can find it at: http://acre.freebase.com/#app=/user/jdouglas/schemas

3) this couldn’t have been possible without Jason’s work upon which I built this so kudos go to him while all the mistakes remain mine

4) I’ve been working on integrating the schema explorer with the upcoming freebase developer’s documentation hub, so that newcomers won’t have to hunt down this URL in the mail archives to find a little guidance in the type maze.

Enjoy!

This release introduces a considerable amount of improvements in both the app editor and in the Acre engine itself.

APIs cleanup and deprecation

Acre APIs have grown organically, starting by emulating a simple CGI-like model. Over time, we realized that a request/response model was better suited for Acre apps.

This Acre release marks the deprecation of the old CGI-like model (for example, acre.environ.* and acre.context.*) and the official introduction of the new request/response objects (acre.request.* and acre.response.*) which are now the recommended way to interact with the HTTP request/response life cycle.

It is important to note that even if deprecated, all APIs are still available and will remain available for a few more releases: we understand that back-incompatible API transitions are highly disruptive and we want to minimize the impact we’re having on our community of developers. At the same time, we want to avoid disruption later and this is why we’re doing this transition now, before a 1.0 release.

It is worth noting that while Acre will execute apps that use deprecated APIs unchanged, it will also log ‘warning’ deprecation messages in the console (which can be obtained by adding setting the “acre.console=1″ parameter in every URL served by Acre).

We highly recommend following those deprecation warnings to update your apps to the new APIs, so that your app will continue to function in the future once the deprecated APIs will be turned off.

The new documentation hub (mentioned below) shows which API methods and variables have been deprecated and what are the new ones that should be used.

App Directory and Documentation Hub

Acre now has two important new visible parts: the app directory and the documentation hub.

• The app directory is the place for browsing and searching for apps. When you release an app to a short url it will be automatically added to the directory. You can edit its description, or even de-list it, in the General tab of the app menu (click on your app’s name) in the App Editor. Find the app directory at http://freebaseapps.com/apps
• The documentation hub is the place where you can find information about all the Acre APIs, the Freebase APIs and web services, the MQL language, etc. It’s still a work in-progress, but we’re also excited to introduce a new documentation viewer that brings together all of our reference documentation — for Acre, MQL and the Freebase Web Services. It’s both available in App Editor from the help menu, as well as stand-alone at http://freebaseapps.com/docs

App Editor

Several important improvements were made in Acre’s app editor:

• new query editor: guides you thru the creation of a MQL query with context-based auto-completion. It will also generate a simple acre template to generate an HTML page from your query and get you going faster. The keyboard shortcut for invoking “query assist” is the Tab key.
• code completion: app editor is now is capable of providing code-completion functionality and guide you during app development thru the discovery of Acre’s API and its parameters. The keyboard shortcut for invoking “code assist” is Alt-Space on the Mac and Ctrl-Space on other platforms. Code assist also shows up automatically after you type a period after “acre”.
• improved app selection menu: makes it easier to move from one app to another with the list of most recently used apps.
• console goodies: the log console now has a link to the source code of the page being executed and a breakdown of the ‘costs’ of executing the request (useful for profiling).
• no more extra blank lines inserted while cutting and pasting code inside app editor.
• files are now automatically saved when previewing.

NOTE: if you see some weird behavior of the app editor, try ’shift reload’-ing it as some browsers are pretty aggressive with caching parts of the application.

Engine

• allow ${} substitutions in <script> and <style> tags
• error pages have links back to the originating source line
• shift-reload now always returns fresh values

For more information on all the bugs fixed in this release, you can look at the issue tracker change log.

Happy Acre coding!

We just pushed a new release of Acre, our hosted environment for developing Freebase apps and services.

Some highlights of the release:

Short URLs
You can now register a short http://*.freebaseapps.com URL for your app in addition to the http://*.username.user.dev.freebaseapps.com/ URL your app was given when you created it. Registering this “release URL” means your app will also be listed in the up-coming App Directory (soon!). To register a release URL for your app, go to the new Versions tab of the app menu.

App Versions
While there’s much to love about the live editing nature of Acre… it’s not without it’s downsides. How do you share your app with people without having to worry about breaking it as soon as you resume development?

To address this problem, we’ve introduced a new versions feature. Versions are addressable snapshots in time of an Acre app. You create a new version of the current state of your app just by giving it a label. That label can then be used in URLs to always refer to that app as of that moment in time, e.g.:

http://label.appname.username.user.dev.freebaseapps.com/

Note that it’s only the top-level app that is frozen in time. If your app refers to files in other apps, they do not inherit the same timestamp. (If you’re adventurous, you can try out the new versioned acre.require — there’s now a second argument for the label — but it hasn’t been thoroughly tested yet.)

In addition, any version can be mapped to the app’s release URL if you’ve registered one. So you can create a version, set it as the released version, and then continue developing. Once you’re at a new safe point, you can the create another new version, update it as the released version, and your users should be seamlessly upgraded.

New Debug Console
We’ve made a shift in this release to sending logs back down in HTTP headers (when the x-acre-enable-log request header is present in the request). This change should be transparent other than the fact that the web-based console is now triggered by the presence of an acre.console=1 query arg instead of the ‘/acre/preview’ path prefix we used to use.

console.log to Firebug with FirePHP
The switch to header-based logging also now enables the use of the FirePHP Firefox extension with Acre apps. If you can live with the PHP formatting, this is a great debugging tool as you’ll see all of your console.log statements for every single Acre request… including AJAX requests and POSTs.

New App Editor Design
To accommodate all of these new features (and a few more coming soon), we’ve redesigned the App Editor. The redesign better surfaces the most commonly-used editor functions as buttons while still showing more code. We also got feedback that having the App and File menus next to each other was confusing, so the App menu is now separate and triggered by clicking on the app name in the header. There are many more improvements too, like IE7 and IE8 support… so just check it out for yourself.

New Code Editor
We’ve also upgraded the version of the CodeMirror library that we use for editing. In addition to fixing a number of bugs, the current line will now be remembered between file switches. Indentation is also no longer forced. So:

• Tab key now shifts a line to the right rather than re-setting indentation
• Highlighting a section and pressing tab shifts the whole section
• To re-indent a selection, press shift-cmd-i on a Mac and shift-ctrl- i on a PC.

Acre API updates

• acre.freebase.img_thumb(id, height, width, mode, errorid) has two new arguments: mode and errorid. Mode takes any of ‘fit’, ‘fill’, ‘fillcrop’ and ‘fillcropmid’, while ‘errorid’ is the ID of the placeholder image to use if an image is not found.
• acre.form_url(base_url, dict). This is a new helper API that will take a base URL and append a serialized Javascript dictionary as a query string to it. This is basically the reverse of acre.environ.params.

Acre API Reference & Code Search
We’re experimenting with a new API browser (much more current and easy to browse, but still lacking some detail), as well as making all Acre scripts searchable so you can find example uses of different APIs. You can try out both from the Help menu in App Editor or directly at
http://api.freebaseapps.com/ and http://codesearch.freebaseapps.com/.

Faster JSON parsing
All of our JSON parsing used to be done in Javascript, which for large requests (like, say… a big mqlread :-) could get quite slow. In this release, we’ve switched to a native Java JSON parser which is much faster.

• acre.response object. With this release, you can now set headers incrementally and from within templates using the new acre.response object. Previously, the HTTP headers of the response could only be set all at once via acre.start_response. In addition, acre.start_response was only usable from scripts and could not be called from within templates. While scripts that currently use acre.start_response should continue to work for now, we do plan on deprecating that API in the future. More documentation here.
• Easily pass variables to the browser. To make it easier to send a value from a script running in Acre to a script running client-side in the browser, we have added a new convenience method called acre.markup.define_browser_var(value, name). This new API just outputs a script tag which declares var name = value. This API is necessary because it’s not possible to safely support $-substitution within <script< tags.
• HTML and XML parser changes… and Sizzle support. These have been moved to acre.xml and acre.html (form XML. and HTML respectively) in order to avoid some conflicts we were having with Rhino’s internal parsing. In addition, thanks to Stefano, there’s now an example of how to use Sizzle (jQuery’s DOM selector engine) server-side in Acre. If you have a use case involving web-scraping, then you’ll want to take a look: http://sizzle.stefanomazzocchi.user.dev.sandbox-freebaseapps.com/
• App Editor now supports IE7. Finally, some support for that “other” browser. :-) We haven’t had a chance to test it as thoroughly as we’d like, so please let us know at bugs.freebase.com of any issues you find. IE6 is still not supported (and won’t ever be) and we’re waiting for the official release of IE8 to complete support for that one… although it should work OK in IE7 mode.
• New API Keys dialog in app editor. If you want to add or remove API Keys (secrets for using third-party services that you don’t want others to be able to see or clone in your source code) you can now do that from within the app editor itself by picking API Keys… from the App menu.
• More oauth providers. We’ve added built-in support for Google and Netflix. Because Google requires oauth credentials to have the scope of a specific service, we’ve defined separate providers for each to make it simpler to use. You still only need to enter in an OAuth token and secret once for all of the Google services (in the API Keys dialog), however.
• Caching off by default. We used to send somewhat non-sensical cache control headers by default. We’ll be trying to set more intelligent defaults over the next few releases, but for 17 the default is to not cache at all. However, you can use acre.response.set_header to explicitly set your own cache control headers if you want.
• Customize your own error page. Want to provide something more friendly than the “fail rhino” to your users? If you create a file named ‘error’ in your app that will be displayed instead of the default error script. The error information will also be available in acre.environ.error_info if you’d like to include that in your page.

Anyone with an intermediate-level understanding of HTML and JavaScript can start right in with building Acre applications, but we’ve also developed a couple tutorials to show you how it’s done and introduce the Acre API and templating commands. The first one shows how to build an application that retrieves information from Freebase using a MQL query and then display the results. The second one takes the static features of the first tutorial a step further, by showing you how to create a user input that overwrites the MQL query, then display images associated with the results and click through to Freebase topics. On top of these there’s also a tutorial on how to build a Facebook app using Freebase data.

In the next week or so we expect to publish some more tutorials, including one on how to incorporate the Freebase autocomplete functionality in your applications, and another on how to create an app that writes to Freebase. If you have any questions, feedback, or comments about these or other Acre tutorials, be sure to let us know!

The first and most important feature is the ability to write to Freebase, enabled by OAuth and the API keystore. We talked about this at our Hack Day back in November, and we’ve had a kind of rudimentary write support for those who wanted to experiment, but now you’re fully able to write apps that write Freebase data on behalf of any logged-in user, like data entry tools, games, and more.

What’s particularly interesting about OAuth and the keystore is that you can use them to connect to other providers, such as Yahoo and Google, that provide OAuth authentication. You can even write to those services using OAuth. And the API keystore can be used to store arbitrary API keys, not just OAuth secrets, so you can use it to connect to sites like Flickr, Facebook, or Amazon.

Further reading: More detail on writing to Freebase with Acre, Freebase OAuth support, and the Keystore.

New Acre APIs

In addition to new APIs for the keystore and OAuth, there is also console.log which allows for full object dumping, and new APIs for using Freebase services (including search and user_info, which did not previously have helper APIs).

Further reading: Acre API reference

App Editor improvements

App Editor has had a ground-up rewrite that is significantly faster, has a new design, and has a couple of nice new features thrown in for good measure:

• New preview window with a Firebug-style object browser for things logged with console.log
• A new ‘App’ menu for moving, deleting and changing the authors of an app (none of which was there before).

Further reading: See all bugfixes and improvements in this release.

Find out more at our user group meeting

We’re having our semi-regular San Francisco user group meeting next Tuesday, Jan 27th, and members of the Acre team will be demonstrating the new features and taking Q&A. Please attend! RSVP via upcoming.org.