I’ll have the DOMFileSystem with a side of read/write access please…

Filesystem access has been a pipe dream for web developers for many years. With the ever evolving complexity of web apps and their need to potentially process large amounts of data, filesystem access is the next evolutionary step in order to push web apps to the next level. Thankfully, smart people have been thinking about these issues and defining new and useful specifications that fill those gaps. Eric Uhrhane of Google has been working on the working draft of the File API: Directories and System specification which defines a set of APIs to create a sandboxed filesystem where a web app can read and write data to.

A recent release on the Chrome Dev channel shipped early support of this new specification allowing us to play around with the new API.

Update: The API is now prefixed with webkit for File System Access and the BlobBuilder, see announcement.

Update 2: requesting PERSISTENT storage needs to go through the Quota Management API, I’ve updated the demo and source files to use TEMPORARY instead, to use PERSISTENT see HTML5Rocks article.

Isn’t this what indexedDB/WebSQL, applicationCache and WebStorage is for?

What this API offers over indexedDB/WebSQL, applicationCache or WebStorage is the ability to process, store and organise very large amounts of data, such as videos. The specification mentions some potentially useful use cases where this API would be advantageous. The developer has the ability to store these large files in a sandboxed environment. If the browser closes or crashes we can access the previously loaded files from our sandboxed filesystem.

However the WebSQL and indexedDB both have the ability to store “Blob” objects which could potentially be used to polyfill support for the Filesystem API. Although indexedDB Blob support has only just landed in Firefox 11.

If you don’t have Chrome Dev installed you can watch the screencast below showing the demo in action. If you do have Chrome dev installed you need to launch chrome with this flag –unlimited-quota-for-files.Chrome no longer needs to be started with this flag for the FileSystem APIs to work! If you’re not running the demo in your localhost but from a file:// protocol you’ll also need to do the additional –allow-file-access-from-files flag for filesystem access to work.

The above demo has a few buttons which run some code to create files/directories or write text to a file. You can also drag and drop files which will be copied into your new sandboxed filesystem.

Requesting access

In order to gain access so we can read and write to our filesystem we need to request access which will create our filesystem based on a few parameters we pass to the requestFileSystem() method.

window.webkitRequestFileSystem( TEMPORARY, 5242880 /* ~5MB */, function(fs) {
    fsys = fs;
    root = fsys.root;
}, FSA.error);

On the window object we have the requestFileSystem() method that takes four arguments (type, size, successCallback and errorCallback), the last one being optional.

  • type: Can either be PERSISTENT or TEMPORARY either it needs to stick around or we only wish to use it for a while and don’t care if the browser needs to remove it.
  • size: How much storage we need, in bytes, for our filesystem. This is ignored at the moment since we’re setting the –unlimited-quota-for-files when launching Chrome.
  • successCallback: Upon successful creation or successfully accessing a previously created file system we can run some code. In the above example I store the DOMFileSystem object so we can access it later.
  • errorCallback: If something goes wrong such as requesting too much storage space we can capture the error and display a useful message to the user.

In the specification it mentions asking permission from the user to create a filesystem. As of writing it never asks the user for permission but I imagine this will eventually trigger a permission request much like geolocation and web notifications currently do.

Creating files and directories

We now have access and have created a reference to our DOMFileSystem inside our immediately-invoked function expression, (function(){})(). This way we don’t pollute the global scope and can have private variables only accessible within the execution context. Let’s dive into creating files and directories.

Create a file

In order to create a file inside our DOMFileSystem we have the getFile() method, which is oddly named for creation but it will make sense in a second.

FSA.createFile = function(path) { 
    root.getFile(path, 
        { create:true, exclusive:true }, 
        function() { 
            console.log(path+ " file created"); 
        }, 
        FSA.error ); 
};

Inside the FSA.createFile() method we pass a path argument and access our DOMFileSystem using root variable we created on the inital filesystem request. On root we have the getFile() method which takes four arguments (path, flags, entryCallback, errorCallback) , the first being the only required one.

The flags object has two properties, create and exclusive. In the above code example I specify both create and exclusive to be true which means if the file doesn’t exist create it and if it already exists throw an error. If we don’t specify create and the file doesn’t exist it will trigger the errorCallback if specified. The entryCallback will let you add data to the current file, which I’ll go into later in article.

Create a directory

Creating a directory works exactly the same as creating a file except we have the getDirectory() method which takes the same arguments as getFile().

FSA.createDirectory = function(path) { 
    root.getDirectory(path, 
        { create:true, exclusive:true }, 
        function() { 
            console.log(path+ " directory created"); 
        }, 
        FSA.error ); 
};

Within the path argument we can specify a relative path but we can’t create a folder more than one folder deep if none of its parents exist e.g. setting path to “multiple/levels/deep” will fail if “multiple” and “levels” folders don’t exist.

If the folder structure root > multiple > levels exists we can then create nested folders easily by passing our path argument as “multiple/levels/deep”. We can also get fancy using “..” in our path argument e.g. If I want to create a folder at the same level at “levels” I can pass “multiple/levels/../samelevel” to the path argument and get “levels” and “samelevel” inside “multiple”. This concept also applies to getFiles().

Writing to files

getFile() and getDirectory() also have the ability to write content to a file or create a file within a directory. This is done through the successCallback which passes in either the fileEntry or directoryEntry reference so we can further manipulate it.

Write content to a text file

In the callback when we have access to the fileEntry which has two methods we can use, createWriter() and file(). To write to a file we utilise the createWriter() method.

function(fileEntry) { 
   fileEntry.createWriter(function(writer) {  // FileWriter 

       writer.onwrite = function(e) { 
           console.log('Write completed.'); 
           dir = null; 
       }; 

       writer.onerror = function(e) { 
           console.log('Write failed: ' + e); 
       }; 
        
       var bb = new WebKitBlobBuilder(); 
       bb.append(data); 
       writer.seek(writer.length); // Always append text to end of file. 
       writer.write(bb.getBlob(mimetype)); 

   }); 
}

In the getFile() successCallback we initialise the createWriter() method on the current fileEntry, attach a few listeners to the writer, onwrite and onerror so we can trigger some code when it’s done or if something goes wrong. There are several other events that are currently not listed in the spec:

  • onabort
  • onprogress
  • onwriteend
  • onwritestart

To append the data to the file we create a new BlobBuilder, this is part of another File API extension specification called File API: Writer. We append our passed in data to the blob, we then set the seek() method to the writer length so we append our text to the file rather than insert it over the top of existing data. Using the write() method we then get the blob data to write to the file.

Getting fancy, drag and drop to your filesystem

The above example may have looked over complicated for just adding some text to a file but it becomes much more powerful when playing with advanced file creation. If you watched the demo screencast or played around with the actual demo you would have noticed that you can drag and drop files to the drop area and it will add those files to your sandboxed filesystem.

FSA.handleDrop = function (e) { 
   var dt = e.dataTransfer, 
       files = dt.files, 
       count = files.length, 
       file, fileType, 
       droppedFileName; 
    
   e.stopPropagation(); 
   e.preventDefault(); 
    
   for (var i = 0; i < count; i++) { 
       file = files[i]; 
       fileType = file.type; 
       droppedFileName = file.name; 
        
       console.log("File type: " + fileType + "n" + "File name: " + droppedFileName); 
        
       dropReader = new FileReader(); 
       dropReader.name = droppedFileName; 
       dropReader.type = fileType; 
        
       dropReader.onloadend = function(e) { FSA.write2File.call(e); }; 
       dropReader.readAsArrayBuffer(file); 
   } 
};

In order to create copies of your dropped files we read in each file, using the File API, as an ArrayBuffer. ArrayBuffer support is recent addition to the File API spec. That way we can pass in the ArrayBuffer to the BlobBuilder, which also had the recent addition of accepting ArrayBuffers, and write it to the file using the Blob data. This allows us to handle just about any file.

That was fancy, but let’s get really fancy

All the interaction in the demo has been creating files from either in the demo or adding locally available files through drag and drop. How about adding files pulled down from a web server. Better yet how about pulling down a tarball file through an XHR request, parsing and extracting its contents and adding all the files to your sandboxed filesystem? Sounds crazy but we can do that.

Using the very clever MultiFile tarball parser by Ilmari Heikkinen and the new Uint8Array type to convert the returned binary data to a typed array. We can accomplish just that.

MultiFile.stream(name, function(f) { 
   var name = f.filename, 
       data = f.data, 
       mime = "application/octet-stream", 
       ui8a = new Uint8Array(data.length), 
       bb = new WebKitBlobBuilder(); 
    
   for (var i = 0; i < ui8a.length; ++i) { 
       ui8a[i] = data.charCodeAt(i); 
   } 
    
   bb.append(ui8a.buffer); 
    
   FSA.write2File(name,bb.getBlob(mime),mime); 
});

Using the MultiFile.stream() method we can stream each file extracted from the tarball and run a callback function. We then convert the raw binary to a typed array, append it to a blob as an ArrayBuffer using the buffer attribute (thanks Eric Bidelman for the tip) and pass of the information to the FSA.write2File() method. This takes the blob and writes it to our sandboxed filesystem.

XMLHttpRequest Level 2 specification does have the responseBlob attribute defined which would allow us to avoid the memory intensive operation of looping over the binary bits to convert it to an ArrayBuffer. Instead we could append the response straight into the BlobBuilder. Unfortunately Chrome doesn’t yet have this response type so we have to resort to the above method

This opens up some really great possibilities for some really powerful web apps of the future. Ilmari even has a gzip parser so we could also stream a compressed tarball.

Iterating over directory contents

In the demo if you’ve added any images by dragging and dropping them. You can click the “get images” button which will iterate over the root directory and find any images and append them to the inside the drop area. This is also executed on window load so any previously added images will be displayed onload.

FSA.getFiles = function() { 
   var dirReader = root.createReader(); 
    
   dropContainer.innerHTML = ""; 
    
   dirReader.readEntries(function(files) { 
       for (var i = 0, len = files.length; i < len; i++) { 
           if(files[i].isFile) { 
               FSA.getFile(files[i].name); 
           } 
       } 
   }, FSA.error); 
};

In order to iterate over the root directory we use the createReader() method which creates a new DirectoryReader. Then we use the readEntries() method to read each entry. Using a for loop we go over every file check the isFile flag to make sure we’re dealing with a file, directories have isDirectory, and then pass of the file name to the FSA.getFile() method.

FSA.getFile = function(name) { 
   root.getFile(name, null, function(fileEntry) { 
       fileEntry.file(function(file) { 
            
           if("createObjectURL" in w && img.test(file.type)) { 
               var url = w.createObjectURL(file), 
                   image = document.createElement("img"); 

               image.src = url; 
               image.width = 300; 
               image.height = 200; 
                
               dropContainer.appendChild(image); 
           } 
            
       }, FSA.error); 
   }, FSA.error ); 
};

In the above code we pass the name to the DOMFileSystem getFile() method, inside the success callback we access the file in question, test to make sure createObjectURL is supported and that the file is of an image type.

createObjectURL() allows us to avoid loading each file into memory and rather generate a blob URL that points to the file. So rather than setting the source of an image as a base64 string we set it as the blob URL and it works like referencing a normal image url.

Removing directories and its contents

If you need to remove any directories and all its containing files you can use the removeRecursively() method on a directory entry.

FSA.removeDirectory = function(path) { 
   root.getDirectory(path,null,function(dir) { 
       dir.removeRecursively(function() { 
           console.log("'" + path + "' and all its contents was succesfully removed"); 
       }, FSA.error); 
   }, FSA.error); 
};

Basically you call the getDirectory() method and in its successCallback use the removeRecursively() method which takes two arguments; successCallback and errorCallback. Trying to delete the root directory will throw an error as you cannot do that.

Removing individual files

In the spec on a fileEntry there is a remove() method, as well as several other methods like copyTo(), unfortunately these aren’t available in the current dev build of Chrome. You cannot delete individual files at the time of writing.

Handling Errors

In my demo I set all the error callback arguments to my FSA.error() method this allows for easy logging of errors in a central place.

FSA.error = function(e) { 
   switch(e.code) { 
       case 1: 
           console.log("File doesn't exist"); 
           break; 
       case 2: 
           console.log("Stop being silly"); 
           break; 
       case 9: 
           console.log("You've already created it"); 
           break; 
       case 10: 
           console.log("You've either exceeded your storage quota..."); 
           break; 
       default: 
           console.log(e); 
   } 
};

Inside this function I have a switch statement that lets me fork for certain errors. You can then either do what I’m doing and log a message to the console or run another function. You see a full list of FileErrors and what number means what.

Also check out the Peephole Chrome extension which makes it very easy to debug the FileSystem API.

Can I use it?

Not really, browser support is Chrome Dev and you need to launch it with a flag in order to use it. This is very early days but it’s exciting to be able to play with the future APIs in the browser and get a real feel for them.

If it’s Chrome only, sure, it’ll be fine to use to add some extra functionality to your web app. You could fallback to WebSQL/indexedDB Blobs for other browsers.

I’ve only scratched the surface of what is coming. I’m sure this article will become out of date as the discussion progresses for the API. I’ll try to keep pace with it. You should always refer to the specifications for the latest information.

Short URL: http://cssn.in/ja/030

 

Post filed under: javascript.

Skip to comment form.

  1. GDmac says:

    And (zipping) uploads from the directory to the server?

  2. Dmitriy Pashkevich says:

    Great insight on the upcoming technology! Thanks!

  3. Hmm, interesting on the other hand if a lot of websites started to use it that would waste space on the hard-drive very fast.

    • @WDM – I guess if storage becomes an issue on the clients computer the API will start throwing exceptions about storage space.

      At the moment there is no permissions based system for this but once the flag is removed it will require user interaction in order to give the web app access to a sandboxed filesystem.

  4. I stumbled across this API tabbing through the JSConsole in chrome in the global scope and consequently, your post. I had no idea there was such extensive support in Chrome already for all this stuff. I can already think of a whole host of screenshotting JS webpages you could build with this….

    • @Adam C – Yeah it’s pretty surprising this landed so quickly but I guess that’s exactly what the dev and canary builds are for. I wrote a little wrapper API to make it easy to request access and read/write files using a convenient callback system. BTW you original comment came through fine. I deleted the other two.

  5. Techno says:

    Is this insecure in any way ? I hope it’s not.

    • @Techno

      It’s pretty early days at the moment. The filesystem is sandboxed from your native filesystem and it will have permissions similar to how geolocation currently works.

  6. rads says:

    this demo does not work for mozilla ,throws exception filesystem not supported and root parameter is undefined

  7. rads says:

    does this demo works for firefox ? If it requires any plug-ins ? For me it is not working neither for chrome nor for firefox.Give me suggestions for it,any help will be highly appriciated.

  8. imtiyaz says:

    I need to zip the sandbox folder and download it to desktop.
    How do I achieve it?

    Thanks, Imtiyaz

  9. Hermann says:

    Thanks for such a great article.

    But in Chrome 21, I no longer find the actual file directory that stores the files from fileSystem in C: . How can I decode it so as to get the files?

    • Ryan Seddon says:

      @Hermann

      I believe you can no longer just navigate to where the files are on your system, I’m pretty sure they’re stored with random names in hidden files/folders within the Chrome directory.