Some love for JavaScript applications

Posted by Victor Costan on August 31, 2012

During our last hack week, Aakanksha Sarda and I set out to build a library that helps JavaScript developers use the Dropbox API. My main goal was to take “static” Web applications to the next level. However, the JavaScript library can be useful for any application that runs a significant part of its logic in the browser, as well as for node.js server-side code.
If you prefer getting your hands dirty right away, go ahead and register for a Dropbox API key, set up an application in your Dropbox, borrow sample code, and read the library documentation.

Motivation: static Web applications

Thanks to recent improvements in browser support and VM performance, I often find myself writing small and medium applications completely in JavaScript, whenever I can get away with it. JavaScript runs on users’ browsers, so all the application’s files are static, and can be served by any plain old file server such as nginx, pretty much any Web hosting service, and your humble Dropbox.

My interest in static Web apps is greatly influenced by how easy they are to deploy. For example, the deployment process for my Dropbox-hosted applications is the cp command (copy if you’re on Windows). Dropbox syncs the files to its servers, and even lets me revert a bad deployment. I’ve been using this beautiful, simple paradigm for all my applications that don’t need to store per-user data.

However, up until now, having to handle per-user data has been a different story. I needed a database to store the data and an application server to talk to the database server. My applications then needed accounts and authentication to ensure that users wouldn’t overwrite each others’ data. Finally, deploying a new version of the application involved pulling the updated code from a git repository, migrating the database schema, and restarting the application server. I was afraid I’d make a mistake, so I ended up writing complex scripts to handle the deployment.

To my despair, the code for having small applications store per-user data was much longer than the actual application code. Deploying the apps was a problem on its own, and left me yearning for the minimalistic “copy to Dropbox” approach. Fortunately, today’s announcement fixes everything!

Demo: “powered by Dropbox” Web applications

My hack week project, dropbox.js, makes it easy to use Dropbox for storing per-user data. For example, let’s consider Checkbox, a To Do manager hosted entirely in this Dropbox folder. While Checkbox won’t be winning design or usability awards any time soon, it is fully functional! It can store your To Do list right in your Dropbox, in less than 70 lines of HTML and less than 300 lines of commented CoffeeScript (which compiles into less than 350 lines of JavaScript).

Let’s skim the Checkbox source code. The action happens in the app’s CoffeeScript code. The Checkbox class is the application’s view and controller, so it renders the UI and handles DOM events. The Tasks and Task classes implement the data model and the Dropbox integration.

Checkbox uses the “App folder” Dropbox access level, so Dropbox automatically creates a directory for my app data in my users’ Dropboxes. The data model design favors ease of development and debugging. Each task is stored as a file whose name is the task’s description. Tasks are grouped under two folders, active and done. Operations on tasks cleanly map to Dropbox file operations in Dropbox. For example:

  • to initialize the task database, create two folders, active and done (implemented by the mkdir calls in the load method in the Tasks class)
  • to create a task named “Buy milk”, create the empty file “active/Buy milk” (implemented by addTask in the Tasks class)
  • to mark a task as completed, move its file from the active folder into the done folder (implemented by setTaskDone in the Tasks class)
  • to remove a task, delete its file from the user’s Dropbox (implemented by removeTask in the Tasks class)
  • to list all the user’s tasks, read the contents of the active and done folders (implemented by the readdir calls in the load method in the Tasks class)

Most importantly, each Dropbox operation takes up one line of code in the Tasks implementation. Thanks to dropbox.js, the Checkbox source code is all about the application’s logic, and does not get distracted with infrastructure issues.

In the end, dropbox.js makes it easy for me to store my users’ data in a system whose features and reliability go beyond those of the storage backing many enterprise-grade applications. Checkbox-managed To Do lists are transmitted securely, stored redundantly, and backed up. My own To Do lists are in my Dropbox, so I debugged my application with my file manager, and deployed it using cp. I have never SSHed into a server, and I haven’t issued a single SQL query.

Want in on the action? This little JavaScript application can help you run small Web applications out of your Dropbox!

dropbox.js: designed to make developers happy

Hopefully, I’ve convinced you to try out dropbox.js for your next prototype or hobby project. In fact, I won’t be offended if you stop reading now and start using it! However, I think you’ll also find it interesting to read more about the goals, tough decisions, and story behind the library’s design. Let’s start with the goals:

dropbox.js stays out of your way

I think that infrastructure libraries should silently support your development efforts, and stay out of your way. Every bit of brainpower lost on figuring out low-level issues is not spent on making your application awesome. So the overarching goal for dropbox.js is to have you spend as little time as possible thinking about it.

To that end, the libraries aims to follow the principle of least surprise, and to make the common use cases especially easy to implement. For example, the interfaces of the methods for reading and writing Dropbox files heavily borrow from fs.readFile and fs.writeFile in node.js, as you can see in the code below.

  1. // Inefficient way of copying a file.
  2. client.readFile(“source.txt”, function(error, data) {
  3.     client.writeFile(“destination.txt”, data, function (error) {
  4.         console.log(“Done copying.”);
  5.     });
  6. });

Dropbox has more features than your average filesystem, such as revision history, and these advanced features can be accessed by passing an options object. For example, the same readFile method can be used to retrieve an old revision of a file.

  1. client.readFile(“source.txt”, { versionTag: “0400000a” },
  2.             function(error, data) {
  3.     console.log(“Done copying.”);
  4. });

However, passing an options object is not mandatory, and the default options reflect the common use case, such as reading the most recent revision of a file. Compare and contrast with the Win32 CreateFile function, which takes 7 parameters.

Last but not least, the library’s interface acknowledges both experienced JavaScript programmers and experienced Dropbox API users. For example, listing a folder’s contents can be done by calling a readdir method, whose interface matches fs.readdir, or by calling a metadata method, whose interface is closer to the /metadata REST API.

dropbox.js is easy to hack


I really hope that dropbox.js works out of the box for you, and helps you integrate with Dropbox quickly and painlessly. At the same time, I realize that a small library can’t possibly cover all the use cases, and some of you will have to extend or modify the library.

The library exposes some of its internals on purpose, so you can break the abstractions when you need to. For example, methods that make AJAX calls, such as readFile and writeFile, return the XmlHttpRequest object used for the AJAX call. If you want to implement progress bars for Dropbox operations, you can set up listeners for the relevant XmlHttpRequest events. To further encourage extension, the library’s internal methods are fully documented using JSDoc, just like the public APIs.

Asides from source code, dropbox.js includes an automated build script, and a mostly-automated test suite with very solid coverage. If you need to modify the library, the README file will help you use the test suite to verify the correctness of your changes and incorporate your changes in a minified library build or an npm package. The entire library is hosted on GitHub, so you can easily submit your patches and not get stuck maintaining a fork.

Tough decisions

Perhaps the most delicate aspect of integrating with a Web service is the user authentication piece. Dropbox authentication uses a four-step process that bounces the user between the application’s Web page and the Dropbox servers, so applications will most likely need to customize the process. dropbox.js defines an authentication driver interface for the custom code, and also includes three implementations that will get you going through the prototype stage of your application. For example, the code below covers both library initialization and authentication.

  1. var appKey = { key: “api-key”, secret: “api-secret”, sandbox: true };
  2. var client = new Dropbox.Client(appKey);
  3. client.authDriver(new Dropbox.Drivers.Redirect());
  4. client.authenticate(function(error, data) {
  5.     if (error) { return showError(error); }
  6.     doSomethingUseful(client);  // The user is now authenticated.
  7. });

While writing the JavaScript library, I struggled a lot figuring out how much I can diverge from the Dropbox REST API, in the name of offering a more intuitive API. While I didn’t want to create a RequestProcessorFactoryFactory-like monster, I also wanted to hide some aspects of the REST API that I found confusing.  For example, the JavaScript library wraps the /metadata output, and uses the stat name for the concept of metadata, to match existing filesystem APIs. The rev parameter that shows up in many API calls is renamed to revisionTag in dropbox.js, to avoid the misconception that its value would be a sequential integers, like Subversion revisions. I look forward to seeing how this decision plays out, and to learning from it.

When I started working on the JavaScript library, I envisioned that the Dropbox object would implement a high-level API, and developers would only call into Dropbox.Client if they would need a low-level API. I had high hopes for the high-level API. It was supposed to provide File and Directory classes, automatically cache Dropbox data in IndexedDb, and automatically kick off the authentication process on token expiration. Unfortunately, by the end of the hack week, the low-level API was barely completed, so the current dropbox.js release ships an empty Dropbox object, and the code samples use Dropbox.Client. Ship early, ship often :)

The story of dropbox.js

dropbox.js was designed and developed during the most recent Dropbox hack week, a hackathon where we put our regular work on hold for 5 days and get to work on our crazy ideas and pet projects.

I got the idea to write a JavaScript client for the Dropbox API when I brainstormed for hack week projects, and I realized that all my ideas would be best prototyped as JavaScript applications. At first, I had many doubts about the idea. Would other people use this? Would I be able to make a decent API? Fortunately, I shared my idea with a few colleagues, who encouraged me to commit to the idea and post it to the company’s internal list of hack week projects.

After posting the project, I received an amazing (and humbling) amount of support from fellow Dropboxers. Before hack week, Chris Varenhorst added CORS headers to the Dropbox API responses, which allow dropbox.js to work in the browser.

During hack week, Aakanksha Sarda fleshed out the code for all the file operations, figured out how to deal with binary files (e.g., images), got the automated test suite running in the browser, and wrote Dropstagram, a Dropbox-powered photo-editing application, using WebGL shaders. A few other Dropboxers used dropbox.js and gave us great feedback and a sense of urgency. Rich Chan built a true Internet Terminal running on JavaScript and Dropbox and a stunning visualizer for the revision history of any text file in your Dropbox. Franklin Ta prototyped a Google Chrome extension that lets you download and upload files straight into / from your Dropbox. David Goldstein worked on a secret project that will make Dropbox apps even more awesome, and used dropbox.js to write a browser-based .zip unpacker for your Dropbox files.

Hack week provided an amazing backdrop for the development of dropbox.js. Graham Abbott set up a pod for everyone working with dropbox.js, so it was very easy for us to collaborate and exchange ideas and feedback. Jon Ying sprinkled some design magic on the applications that we developed. The kitchen staff stayed up and prepared hot food for us every night of the week. Dropboxers (including Drew and Arash) came by our pod, looked at our demos, and cheered us on.

After hack week, Chris Varenhorst, Dima Ryazanov, and Brian Smith helped me work through some last-minute technical difficulties. Jon Ying gave a makeover to the Checkbox sample app. Albert Ni, Alex Allain, Glara Ahn and Jon Ying helped make this blog post happen.

Go forth and hack!

I hope you enjoyed reading about the dropbox.js. Above all, I really hope that you will go download the library, and build something amazing with it!

I expect that “powered by Dropbox” Web applications will become a great tool for learning Web programming and building class projects. I’ll walk the walk myself this coming winter, when I’ll teach Web Programming at MIT.

I look forward to learning about the projects that come out of the next Dropbox hack week, and I’ll be counting how many of them use dropbox.js :)

  • http://twitter.com/AwesomeAPI AwesomeAPI

    Nice article and Hack Victor! .. When I get some spare time I will play and see what potential there is… must admit got my interest :) 

    • Anonymous

       Thank you for your kind words! I hope the library works for you. If you run into any issues, please let me know!

  • http://tracker1.info/ Tracker1

    is it possible to have a self hosted js app that other people could self host without an api key.  I was thinking that an iGoogle like app that a user can run themselves could work well.

    • Anonymous

      Thank you for your comment!

      The feature we just released lets you host your app in your Dropbox, and the app gets to write to the users’ Dropboxes.

      I think you can use that to build an iGoogle-like app — have a .json file in the user’s Dropbox listing out all the user’s widgets, plus layout information, then have a .json for each widget’s data. Your app would be a page with many iFrames, and use postMessage to coordinate between them and write to the user’s Dropbox. You could have a catalog of widgets in your Dropbox, and let users add their own widgets as well.

      Can you please help me understand what design you have in mind?

      • Vikram Bhatla

        is it possible to show the file in browser which we is using client.readFile.
        I am getting the binary data which i can load in all the html5 supported browsers(FileReader). I cant show the file using normal browser.

        I actually want to show the dropbox file in a viewer?
        Is it possible to access Dropbox viewer in 3rd party application?
        Can you suggest me what design I can use for displaying the dropbox files in my app?

        • http://blog.costan.us/ Victor Costan

          Sorry I missed your comment earlier, Vikram!

          You can generate URLs to the Dropbox file previews using makeUrl. (see https://github.com/dropbox/dropbox-js/blob/master/src/client.coffee#L428 )

          If you go down that path, you’ll have to redirect your users to that page. At the moment, the Dropbox API doesn’t support any integration with the built-in file preview.

          I hope this helps!

  • http://www.carvermediagroup.com/ Custom Web Applications

    Really nice work. Thanks for sharing this helpfully blog.

  • Jeremy Ruston

    I notice that the Dropbox app key and app secret are both exposed in the source code of the checkbox application. I’m curious whether that’s a problem – doesn’t it mean that someone else could impersonate the app?

    • http://blog.costan.us/ Victor Costan

       Thank you for your interest in dropbox.js, Jeremy! In a nutshell: yes, the API key and secret are exposed. This cannot be avoided for applications that run on the client-side, so you shouldn’t worry about it. The Dropbox authentication model takes this issue into account.

      Slightly longer answer: https://github.com/dropbox/dropbox-js/issues/6

    • http://blog.costan.us/ Victor Costan

      Release 0.6.1 of dropbox.js has support for hiding the API key and secret of your application.

      Relevant doc snippet: https://github.com/dropbox/dropbox-js/blob/master/doc/getting_started.md#browser-and-open-source-applications

      Please use this method (or your own method, if you prefer) for concealing your API credentials before applying for production status.

      • Jeremy Ruston

        Hi Victor – many, many thanks, that’s great, I’m just about to resubmit the app…

  • http://www.multiinfotech.com/ SEM

    Really an awesome post.

  • http://www.multiinfotech.com/ SEM

    Thanks..

  • http://twitter.com/nathancarter5 Nathan Carter

    This API lets me write to other people’s Dropboxes.  Is there an API that lets me write to my own?  This would let me, say, store global high scores for a web game in a little folder in my dropbox.  Far easier than writing a server-side script and finding hosting for it.

    • http://blog.costan.us/ Victor Costan

      Sorry I didn’t see this earlier, Nathan!

      The Dropbox API doesn’t have a good model for achieving that yet. The dropboxers working on the API are aware of this use case, and will figure it out at some point.

      If you’re doing a prototype, you can consider setting up a Folder application (please do _not_ do this with a Full Dropbox app), and embedding your own token into the application. This approach has the downside that a malicious user can wipe or change all the score data. So you might get away with it in a class project or in a workshop.

      If you need this in production, I’d (sadly) recommend taking a look at Heroku and Google App Engine, until the Dropbox API adds support for your use case. Both alternatives have free quotas that should get you pretty far, assuming your code is reasonably efficient.

      • http://twitter.com/guillaume86 Guillaume Lecomte

        What about a readonly access_token generator?
        It should be available on any folder, something similar to “get public link” but with api access.

        • http://blog.costan.us/ Victor Costan

          Given the demand for this feature, I’m sure the API team is working on something. As soon as the Core API documentation for the feature comes out, I’ll figure out how to support it in dropbox.js.

      • Tartin

        This is great if it is possible!
        I am actually in need of such a thing for a class project. I am working on a website prototype that needs to store various small data. Can anyone explain the idea of “setting up a Folder application”? I know nothing about server-side scripts, Ajax, or API. I hope someone can give me a hint!

  • http://twitter.com/prshntvc Prashant C

    Wonderful post Victor! Going to use this to prototype my next app. Thanks a lot

  • Kelt

    hey,this is fun coincidence. i will make html5 app and considered dropbox as a json db storage. so i thought ok lets write js wrapper for dropbox api. obviously you had a same idea few weeks ago. looking forward to test your implementation … cheers. Kelt

    • http://blog.costan.us/ Victor Costan

      @Kelt: It shouldn’t be too painful to use dropbox.js for an approximation of a JSON DB. You can call JSON.stringify and then writeFile for db writes, and JSON.parse and readFile for db reads.

      If you want a real database, then you need to worry about concurrency and conflict resolution. That’s not easy, but you should be able to use the Dropbox revision history to do something along the lines of CouchDB’s conflict resolution. Please do let me know if you manage to pull this off!

  • Yogiee

    I second Nathan Carter’s question. Is there a way to use this library to write stuff to my own dropbox folder than user’s?

    • http://blog.costan.us/ Victor Costan

      Please see the answer there. The short answer is “it depends, but probably no.”

  • http://www.t35.com/ Davis Delaney

    Very nice article, i will definitely try this out, simply amazing.

  • http://noelcohen.webstarts.com/ RisaRichardson

    This article validates that there are conscientious writers who do their homework before writing. You have great insight in this subject matter. I respect that.

  • Axon

    Is it possible to save text (like text log or something) directly from browser to a new/existing file in Dropbox folder using JavaScript applet with corresponding permissions? Can anyone show an example?

  • Pingback: Web | Pearltrees

  • http://www.facebook.com/lukeinth Luke Hubbard

    Please add Access-Control-Allow-Headers: * too so we can use headers such as Range (to download parts of files) and If-Modified-Since, etc. Currently I’m stuck without support for additional headers. I dont think there is any security issue in adding support for this.

    • Jesús Leganés Combarro

      I was planning to use the Range header too, so if it’s needed to do some tune-ups on the DropBox servers please do them! :-D

      • http://blog.costan.us/ Victor Costan

        For everyone else reading through the comments here: if you run into roadblocks with dropbox.js, please open an issue on the project’s GitHub page, so I can see it quickly and keep track of it. https://github.com/dropbox/dropbox-js

        @Luke Hubbard, @Jesús Leganés Combarro: Thank you for reporting this issue! dropbox.js is now better because of your input!

  • http://www.mixtapedia.com/blade-brown-bags-boxes-2-mixtape-out-30th-nov/ Blade brown

    Really nice work. Thanks for sharing this helpfully blog.

  • http://brianmhays.com Brian M. Hays

    This is cool