Bitcasa Power Tip: Use Mirroring for large uploads

Summary

In this post I want to talk about Bitcasa and some of the issues the current desktop clients (Mac, 1.1. build 1106, Windows 1.1.0.0) have when copying large amounts of data to the cloud and describe a tip I've found for making the process a little easier and more reliable. The tip is most effective with the Mac client, but is useful for Windows as well. It won't make your upload any faster if you have a slow connection, but it will make using your computer a little easier while you wait.

The Problem

Since coming out of beta, Bitcasa has had a bit of a rocky start with its desktop clients. It wasn't until the latest Mac release (1.1 build 1106 as of this writing) that it was stable enough for me to use regularly. My current primary use case for Bitcasa is as a off-site backup for all my personal media files, home movies, photos, music and such. To do this I've been slowing copying my data, little by little to my Bitcasa Infinite drive. (Note: for various reasons, which I won't get into here, I'm choosing to make a 'copy' of my data rather than use Bitcasa's built in mirrored folder feature directly). Using the normal copy method, I've been forced to limit my upload to chunks of no more than a few GB at a time due to the following issues:

• Long times to copy to the cache. - Depending on the speed of your drive copying data to Bitcasa can take a long time. By this I mean how long it takes before the copy dialog in finder or Windows explorer completes. What is happening here is that Bitcasa is copying your data to its cache location before sending it up to the cloud.
• Filling the cache. - Since the data must be copied locally before sending you can run into problems depending on how much you try to copy at a time. When copying a large block of data you can either start to block if you have a small fixed cache size, or the cache will grow ever larger to accommodate your copy, chewing up disk space.
• High CPU use. - While the copy is going on, Bitcasa also starts encrypting the data to memory and begins to send it. This has the effect of nearly pegging out your CPU depending on the speed of your machine. The CPU usage will drop a bit after the copy is done and then again, once all the data is encrypted, but that can take several minutes depending on the copy size and leaves your machine sluggish.
• High memory use. - This one is potentially the biggest issue with this method. As the data is copied into the cache on your disk, it is also being encrypted into a memory buffer for transmission to the cloud. In practice this memory buffer is the same size as the data you are copying. So if you are not careful you can use up all your available RAM on a big copy and really bring your system to a crawl.
• Can't throttle or pause and upload. - Once you start the copy process there are no controls currently for managing the bandwidth used for the upload. Bitcasa will typically peg out your upload capacity for as long as it takes. While this is going on anything else you might need to do like commit a large code change or upload a demo video to YouTube - will have to fight for bandwidth.
• Need to babysit the uploads. - For the reasons outlined above you are currently best served by breaking your uploads into smaller chunks. That means you have to keep checking to see when they are done so that you can start another one. Although it doesn't happen very often now, in the past Bitcasa was known to occasionally error our on a large upload causing you to have to start it again and in some cases delete a partially updated file first.

The Workaround

I found a trick for my initial data seeding problems on the Bitcasa forum from a user named Vadym Gnatkovskyy. This trick involves a a few extra steps but has served me very well as I continue to load up Bitcasa with all that data I've been hoarding over the years. The trick involves using Bitcasa's 'mirrored folder' feature as a way to get the data initially copied to the cloud, then uses the 'share' and 'de-duplication' features to get that data where you want it when the copy is done. Here are the steps:

1. Mirror the data instead. - Right click (or ctrl click) on the folder you want and choose Bitcasa... Mirror this folder to Bitcasa. Note: that on Mac only the primary drive can be mirrored, so you may have to copy your data to a temp location first. After you are finished with all the steps listed here, you can delete the temp folder.
   
2. When the mirror copy is finally finished, share the mirrored folder with yourself. - This is done by going to the 'Mirrored Folders' location of your Infinite drive in Finder (or Windows Explorer). Right click on the mirrored folder and choose Bitcasa... Send this folder to a friend. Copy the link, and open it up in the browser. Choose 'Add to your Bitcasa Drive. (You may need to log in first). Finally from the share page, click 'delete this link' now that you are done with it so it doesn't hang around.
3. Move the folder to your final destination. - When you shared the folder with yourself above it will show up at the root of your Infinite Drive. Simply open up another Finder or Explorer window and drag and drop the folder to your final location in your Infinite Drive. The folder will be moved instantly with out having to re-copy it.
4. Stop Mirroring the folder. - Go back to the original folder, right click, and choose Bitcasa... Stop Mirroring to Bitcasa.
5. Stop and restart Bitcasa. - For good measure I've found it best to restart Bitcasa after this step just to make sure the mirroring is really shut down.

So why go through all this hassle just to copy files? This has to do with the different way in which Bitcasa handles a 'new' mirrored folder over a normal copy or even a new file in an existing mirrored folder. When you first mirror a folder, Bitcasa is expecting to send a large amount of data and optimizes accordingly. It is also expecting this data to stay static on your drive and so it avoids the unnecessary cache copy and instead send the data directly from the source folder. This eliminates that long step of copying the data to the cache. Also since the data is static, it doesn't bother trying to pre-encrypt the data to RAM, and instead just encrypts it on the fly as it is uploaded. This takes care of the other two major issues with the copy, the high CPU and RAM use. So what are left with is your data getting trickled up in the background with low CPU and nominal RAM use. On Mac you get the added bonus that you can actually pause the mirror upload! When doing a first time folder mirror Bitcasa will add a new option to the menu bar popup that will show you the current upload rate and allow you to pause the upload. You don't currently get the pause option on Windows, but you can exit the client and restart it again later and it will pickup where it left off.

Conclusion

Hopefully the need for this trick is only temporary. Bitcasa is working hard to improve their desktop clients. Features like being able to pause normal uploads and throttle bandwidth use are on the roadmap. They are also very aware the high memory use issue. In the mean time I am having very good luck shoveling up large amounts of data, sometimes 50GB at a time, using this method.