Hypercritical

Hyperspace Update

by
Hyperspace icon on a star field

Two months ago, I launched Hyperspace, a Mac app for reclaiming disk space without removing files. The feature set of version 1.0 was intentionally very conservative. As I wrote in my launch post, Hyperspace modifies files that it did not create and does not own. This is an inherently risky proposition.

The first release of Hyperspace mitigated these risks, in part, by entirely avoiding certain files and file system locations. I knew lifting these limitations would be a common request from potential customers. My plan was to launch 1.0 with the safest possible feature set, then slowly expand the app’s capabilities until all these intentional 1.0 limitations were gone.

With the release of Hyperspace 1.3 earlier this week, I have accomplished that goal. Here’s the timeline for overcoming the three major 1.0 feature limitations:

Here’s an explanation of those limitations, why they existed, and what it took to overcome them.

Packages

A “package” is a directory that is presented to the user as a file. For example, an .rtfd document (a “Rich Text Document With Attachments”) created by TextEdit is actually a directory that contains an .rtf file plus any attachments (e.g., images included in the document). The Finder displays and handles this .rtfd directory as if it were a single file.

For a package to work, all its contents must be intact. Hyperspace works hard to handle and recover from all sorts of errors, but in the rare case that manual intervention is required, asking the user to fix a problem within a package is undesirable. Since packages appear as single files, most people are not accustomed to cracking them open and poking around in their guts.

This may all seem esoteric, but there are some kinds of packages that are widely used and often contain vast amounts of data. Let’s start with the big one: Apple Photos libraries are packages. So are iMovie libraries, some Logic projects, and so on. These packages are all ideal targets for Hyperspace in terms of potential space savings. But they also often contain some of people’s most precious data.

For the most part, files within packages don’t need to be treated any differently than “normal” files. The delay in lifting this limitation was to allow the app to mature a bit first. Though I had a very large set of beta testers, there’s nothing like real customer usage to find bugs and edge cases. After five 1.0.x releases, I finally felt confident enough in Hyperspace’s basic functionality to allow access to packages.

I did so cautiously, however, by adding settings to enable package access, but leaving them turned off by default. I also provided separate settings for scanning inside packages and reclaiming files within packages. Enabling scanning but not reclamation within packages allows files within packages to be used as “source files”, which are never modified.

Finally, macOS requires special permissions for accessing Photos libraries, so there’s a separate setting for that as well.

Oh, and there’s one more common package type that Hyperspace still ignores: applications (i.e., .app packages). The contents of app packages are subject to Apple’s code signing system and are very sensitive to changes. I still might tackle apps someday, but it hasn’t been a common customer request.

Cloud Storage

Any file under the control of Apple’s “file provider” system is considered to be backed by cloud storage. In the past, iCloud Drive was the only example. Today, third-party services also use Apple’s file provider system. Examples include Microsoft OneDrive, Google Drive, and some versions of Dropbox.

There’s always the potential for competition between Hyperspace and other processes when accessing a given file. But in the case of cloud storage, we know there’s some other process that has its eye on every cloud-backed file. Hyperspace must tread lightly. Also, files backed by cloud storage might not actually be fully downloaded to the local disk. And even if they are, they might not be up-to-date.

Unlike files within packages, files backed by cloud storage are not just like other files. They require special treatment using different APIs. After nailing down “normal” file handling, including files within packages, I was ready to tackle cloud storage.

In the end, there were no major problems. Apple’s APIs for wrangling cloud-backed files mostly seem to work, with only a few oddities. And if Hyperspace can’t get an affirmative assurance from those APIs that a file is a valid candidate for reclamation, it will err on the side of caution and skip the file instead.

Libraries

In the early years of Mac OS X, there were tragicomic tales of users finding a folder named “Library” in their home directory and deciding they didn’t need it or its contents, then moving them to the Trash. Today, macOS hides that folder by default—for good reason. Its contents are essential for the correct functionality of your Mac! The same goes for the “Library” directory at the top level of the boot volume.

Hyperspace avoided Library folders for so long because their contents are so important, and because those contents are updated with surprising frequency. As with packages, it was important for me to have confidence in the basic functionality of Hyperspace before I declared open season on Library folders.

This capability was added last because the other two were more highly requested. As usual, Library access is enabled with a setting, which is off by default. Due to the high potential for contention (running apps are constantly fiddling with their files within the Library folder), this is probably the riskiest of the three major features, which is another reason I saved it for last. I might not have added it at all, if not for the fact that Library folders are a surprisingly rich source for space savings.

The Future

There’s more to come, including user interface improvements and an attempt to overcome some of the limitations of sandboxing, potentially allowing Hyperspace to reclaim space across more than one user account. (That last one is a bit of a “stretch goal”, but I’ve done it before.)

If you want to know more about how Hyperspace works, please read the extensive documentation. If you're interested in beta testing future versions of Hyperspace, email me.

In some ways, Hyperspace version 1.3 is what I originally envisioned when I started the project. But software development is never a straight line. It’s a forest. And like a forest it’s easy to lose your way. Launching with a more limited version 1.0 led to some angry reviews and low ratings in the Mac App Store, but it made the app safer from day one, and ultimately better for every user, now and in the future.