Using Metadata Effectively in OS X
A Long Time In The Making
Not quite a year ago I wrote a feature on LifeHacker talking about the use of Metadata in place of a folder-based organization scheme. Since then I’ve received many emails and inquiries asking for more on the subject. Inevitably I’ve responded with short answers and the promise of a forthcoming post here on The Apple Blog to satiate all the interested parties. I’m here to tell you, that the time has finally come my friends.
I’ve spent a lot of time tinkering with the system, trying new things, combining tools and workflows, and on and on over the year I’ve been working with the idea. So as I’ve been pondering how to approach the piece I wanted to write here on TAB, I’ve decided that doing it in parts will probably work out best. I’d like to say they’ll come out weekly, but there’s a ton of stuff clogging this brain of mine, and it may be more like bi-weekly, so just an fyi on that.
Enough of the pleasantries, eh? Let’s get started with the first installment of Metadata as a Filing System. In this part I’ll be covering the basis of my idea, the tagging structures and best practices that I’ve found work best (at least for me), and the basic goals of using this system.
Some Background to Get Up to Speed
When Tiger launched in 2005 it brought with it Spotlight for unparalleled searching capabilities. A large factor in this amazing new tech was the ability of the operating system to leverage metadata in files. What is metadata? It’s information about information…Ok, a better explanation: Metadata is the term used for details about a file. For instance, who created the file, when, how large is the file size, and so on.
But the best part came with the addition of Spotlight Comments. Accessible through the Get Info screen of any file on OS X, the Spotlight Comments field gives the user free-reign for creating customized metadata (referred to as ‘tags’ from here on out in this article). Think about tagging in the mainstream sense. Just about everyone has seen it in action with the advent of Web 2.0. Del.icio.us uses it for ‘categorizing’ bookmarks, flickr allows you to tag your images, and many blogs provide tag clouds of the many blog post categories offered within their pages. So we translate that thinking to our files and the way we organize them on our hard drive. 
I’m fairly certain your hard drive is teeming with files, most likely in your Documents folder, just like mine. I’ll venture further, to guess that within that Documents folder you’ve got several more folders - maybe a ‘Letters’ folder for all your correspondences, a ‘txt’ folder for all your txt notes, a ‘work’ folder for business stuff, and so on and so forth. Maybe those folders have more nested within them…you get the idea. You’ve got the typical folder hierarchy that every semi-organized computer user has had since the computer shipped with a hard drive larger than a megabyte.
So what happens when you’ve got a letter that you wrote for a business proposal? Do you file it in your ‘Letters’ folder, or your ‘work’ folder? Or maybe a ‘Letters’ folder within your ‘work’ folder… How do you remember which place you put it in? (I know you could create an alias in the location you didn’t store it in, but humor me here - I’m not that well organized anyway!) Well the options metadata gives us, allow for us to file the letter in a general place and tag with with both ‘letter’ and ‘work’ so we can find it later based on the tag, rather than where we may have put it.
When I first started this project, I moved all the files and folders in my Documents directory, into a folder called ‘pretagging’. From there on out anything that would normally have been filed in my Documents folder hierarchy, was just dumped into Documents and left to marinate with nothing but a metadata tag (or two or three). This approach really forces you to commit to the idea of using metadata as a filing system, because then when you look at your Documents folder (rather than just searching the tag(s) in Spotlight), you just see a heaping mess of files to wade through. Great motivation to make it work, eh? So if you’re planning to play along at home, this is a great way to jump right into the deep-end.
Guidelines Are a Must
So Spotlight makes all this possible because these items are then completely searchable on your drive. Great. But before we get too carried away, we need to decide on a useful tagging structure, or set of guidelines. If you don’t, your tags will end up looking much like my initial foray into del.icio.us bookmarking, where there are 1-off tags everywhere. Not good, and not useful for searching.
It’s extremely important to have a plan in mind before you start tagging your little heart out. You don’t necessarily have to decide on the specific tags you’ll be using, because that will change and evolve over time. But keeping in mind a set of guidelines for how you’ll tag your files is the key. I like to stick to singular terms, ‘letter’, ‘project’, etc rather than trying to remember, “Did I use ‘letters’ or ‘letter’ last time?” Then you either guess right and end up ok, guess wrong, and have a 1-off tag that could ruin your search results later, or you waste more time digging to see what tag you need to match. All of these are time-wasters, and who amongst us has time to waste anymore?
So decide if you’re going to use singulars or plurals for your tags. There will of course be times when a tag you choose is most common in the plural, such as ‘keys’, or ‘taxes’. Just something to keep in the back of your mind.
Another point to consider in your guidelines is how granular you want to get with your tagging practice. So maybe you tag some files as ‘project’, but what happens when you get too many project files from different customers? This is where it gets free-form and up to you on how you handle it. You could do something like ‘project’ and ‘clientname’, or just go for broke with ‘clientnameproject’. Figure out what’s easiest and most comfortable for you.
The last thing, and probably the most important - in my opinion - is a prepending symbol for your tags. The GTD’ers use the @ sign most of the time, so I started with that. But I found that Spotlight effectively ignores the @ symbol and returns less than exact results. As an example, ‘@project’ and ‘project’ will both return the same results in Spotlight. Which is to say that they return anything in Spotlight’s index that contains the word ‘project’. That leaves a lot of wading through stuff to find just what you want. With some experimentation I landed on the & sign as one that Spotlight didn’t ignore. So now a query on ‘&project’ returns only the things that match that syntax exactly - no extraneous data or files that will slow you down. Sure you can do a Spotlight search using quotes around your @project search term, but that’s extra keystrokes, and frankly, I’m too lazy for that. So I use the & sign to prepend my tags. You can play around and find something else that may work for you, or just use the one I found.
These ideas lay a decent framework for your tagging scheme. Obviously what works for me may be different for you, so figure out your own system - if you use these guidelines to do it, you’re much more likely to find success in using metadata for your filing system.
So What’s The End Game?
For me the purpose behind all of this was to see if it was really a good alternative to the traditional practice of using folder hierarchies. I wanted to see if I could still locate my files in this manner, and if I could, was it any faster? So I suppose efficiency is the name of the game when it comes time to measuring your success.
This is where it becomes a gray area on whether or not the metadata hits the mark. If you just use the tools that OS X offers (those being Spotlight and using the Get Info screen for each individual file) then there’s no way this method stacks up. The performance gain comes in using 3rd party apps and utilities. I’ve played with many of these, and tried several combinations to determine which seem to work the best. If you’re even a casual reader of The Apple Blog, you won’t be surprised that Quicksilver is at the hub of these tools.
But as I mentioned at the start of this post, there will be a few parts to this piece. Those tools - and the best ways I’ve found for using them - are topic for a forthcoming article.
Moving Forward
There’s still a lot to come on this topic. I hesitate to say just how many installments I’m planning, because I feel like there’s enough to write up for a handful of posts. But for now I’ll venture 3 more parts to this saga. Minimum.
Amongst these future posts I’ll be covering the 3rd party tools that make the metadata system a really useful alternative, and how to put them into action for you. I’ll be putting together one of my weekly screencasts to demonstrate the system I’ve settled into. I think that will give a good feeling for what this system is capable of if you stick with it and truly own it. And finally (at least for now), I’ll talk about where I’d like to take this metadata filing scheme in the future. Essentially some ideas I’ve had for really automating things so that my computer works for me…But more on all this in coming weeks.
Q And A
There are certainly going to be questions abounding from this article. As always, I’ll do my best to answer them in the comments below. However if the questions tread heavily on forthcoming parts of this piece, my answers will likely be short ones, if at all - I’m not ignoring the validity of said questions, I just feel they’ll be more easily accessible if they’re available in the body of each article (rather than buried in the comments).
In fact, maybe I’ll make each week following one of these pieces, a post full of answers to the questions posted in the comments. I’ll have to think on that, depending on how the reaction to this post goes over. Guess we’ll all be surprised…
Trackbacks/Pingbacks (Trackback URL)
Comments RSSComments
#1 Richard Neal says:I find it a lot easier to use a note-organizing application with tags, so everything’s in one place, although I haven’t exactly settled in on one yet. It’s amazing how Quicksilver can be so integrated into everything you do.
#2 Colin says:I tried the “metadata only” filing method for a while and ended up drifting away from it because it was too difficult to do even modestly complex searches in Spotlight (e.g., (tag1 OR tag2) AND (filetype1–especially if it’s not one of the types that you can do a kind: search for in Spotlight)). I’d be very interested in what kinds of 3rd party utilities are around to simply Spotlight search constructions.
#3 Steve Lang says:Just today I found 2 very interesting freeware/open source apps for OS X metadata- SpotMeta and Punakea:
http://www.nudgenudge.eu/
http://osx.iusethis.com/app/spotmeta
SpotMeta is not as slick, but allows you to create new metadata fields that are recognized in both Spotlight and Smart searches. Not just plain text either- you can create date fields, drop down lists, etc. Punakea uses the existing Spotlight comments field, but adds its own nifty touches too.
Looking forward to seeing this whole series!
Thanks,
Steve
#4 Johan Cloetens says:I know a better solution: http://www.inuron.com/movies-intro.html, it even allows you to have access to your data over the internet, with seamless itunes integration (http://inuron.blogspot.com/2006/11/for-itunes-lovers-only.html)
#5 Johan Cloetens says:Soory about this, url should have been: http://www.inuron.com/movies-intro.html
#6 mistercharlie says:Johan:
Very nice looking app, which I saw some time ago.
But they are charging €29 for a *beta*! What’s more, there’s no demo.
So you have to pony up 30 bucks up front, with no idea what you will get, apart from few video clips.
Can I interest you in some snake oil?
#7 BZ says:Could you show how to do quick and easy tagging with Quicksilver as well as something like Yep (used to be free).
I have been scanning for months, have tons of PDFs, but can’t find an easy way to go through them, tag them and move them so I know they have been tagged. A workflow if you will
#8 Michael DiSefano says:I am very excited about this method of organization. For the past few weeks I have been experimenting with various “notetaking” apps such as Journler, Yojimbo, and Devonthink. For my purpose (getting organized) I think leveraging OSX’s built in metadata support will be a much more elegant solution.
I currently use (depend on?) Quicksilver and I’m really looking forward to future installments.
#9 Guthrie says:I’ve been really wondering about Punakea, and if anyone is using it to create this sort of system. OSX seems powerful enough in many ways that maybe all these PIM applications are…overkill? A few freeware add-ons, and I wonder if I can basically accomplish the same workflow.
I’m also wondering if the next Apple OS is going to facilitate this sort of set up even easier.
#10 Nick Santilli says:Rest assured that Quicksilver plays a large part in the system that I’ve become accustomed to. I’ll show it in graphic detail, so fret not!
Guthrie - Punakea is a nice idea. The problem I have with it is that it wraps several symbols around the Spotlight Comments to denote them for its own purposes. That’s fine, but I’d prefer control over that syntax so I can pare it down, or increase its robustness as I feel necessary.
I’ve tried out most of the other apps mentioned thus far in the comments, but have mostly left them behind. Maybe it’s Quicksilver’s fault, but I prefer minimal invasiveness on the part of my apps - I like everything to feel as built into the OS as possible, so the more I have to deal with a ‘clunky’ interface, the less likely I am to stick with it.
That’s meant as no offense to these apps, they just haven’t really fit the style of workflow I’m accustom to.
#11 Alf Watt says:It’s actualy possable to directly edit the keywords, projects and ratings tags of files using Desk Lamp (http://desklampx.com) which also allows you to save searches and organize your results in several views not avaliable using Spotlight or the Finder.
#12 TheMacThinker says:It is definitely helpful to know a little more of the commands that help getting the most out of spotlight…
—————
http://www.mostofmymac.com
#13 Erik Ohlin says:Great article! I look forward to the next installments!
Has anyone taken a look at SpotlightFS? This is from the Google team and it’s an implementation of the MacFUSE filesystem which dynamically queries Spotlight to create the directory structure.
http://code.google.com/p/macfuse/wiki/MACFUSE_FS_SPOTLIGHTFS
#14 Scott says:At work I am in a multiplatform environment and I am concerned about how spotlight metadata and tags will be perserved when I copy a file to a windows server or send it via email. I often keep old files on our server but I want to be able to preserve the metadata for searching or if I copy it back to my Mac. Do you have any info on this?
#15 Rene says:Is there anyway to add a tag while you save the file in your app? Example:
I create a text doc and save the file as name.txt in some directory. I then need to browse to this directory find the file, click info then add the tag.
Seems like a lot more work then just saving it to a sub directory some where.
I never tried using metadata but is this the process i would need to do?
#16 Teeming says:The word you’re looking for is “teeming” not “teaming”.
#17 Nick Santilli says:Rene -
I’ll cover all that next week, but there is a utility that adds that functionality. Default Folder X
#18 OH NO says:I use Ubuntu >_
#19 OH NO says:Is there a way to do metadata sorting for Linux?
I really like the idea, it’s like Gmail’s labels.
#20 Michael DiSefano says:@ Rene:
I currently use Default Folder X (in demo mode as I think the price of $35 is a bit steep) which allows you to comment right from the save dialog. It does much more than this, but that, I feel is its moste useful feature.
I’m anxiously waiting for the next installments because I would like to stop using Default folder X and if I could replace it with Quicksilver, that would be great, one less startup item!
#21 Michael DiSefano says:looks like Nick beat me to it, sorry for the duplicate post
#22 brock says:For any of this to work for me, the system would have to be a series of keywords that I create/edit and can simply select each time. otherwise, it is too much trouble. I am hoping for something like this in Leopard. If not, folders and good titles for documents work well for me.
#23 weldon says:great post, Nick. Looking forward to the next installment. Maybe you could comment on the other wildcards in spotlight (like @) and what they do in searches.
#24 Alex Williams says:This seems like a great idea for organizing files, except I’m curious to know if the metadata remains with the file after it’s saved (burned) onto a CD or DVD.
If not (if stored in an index file), then I can’t see myself backing up the metadata index each time I want to backup my “Documents” directory.
So this solution might be great now, but if a computer crash were to occur and I needed to restore my files, I would be furious to find out NOTHING is organized anymore.
#25 on the edge of my seat says:Where’s p.122???? geez, way to leave us hanging….
#26 brock says:anyone try Punakea?
http://www.macupdate.com/info.php/id/23401
Something like this could work for me. Only thing is when you open get info window on a file, the comments section looks something like this
###begin_tags###@tickets;@christina;@concert;###end_tags###
Kinda creepy
#27 Lucas says:Does anyone know if it would be possible to write a Folder Action AppleScript that would automatically open the Get Info window for files that you save to your Documents folder? I’m asking because I’m not *terribly* familiar with AppleScript, but I know enough about it to think that it wouldn’t be too tough to write if you had the know-how. With something like that set up, adding tags to your files would become really simple and efficient.
#28 Michael Barber says:Thanks for a nice article. I’ve been considering a similar metadata-centric approach to document management myself, so it is quite encouraging to see that you’ve been successfully doing that for over a year. My own thinking is to use a mix of folders for programmatic work (no way around this) and tags for managing references,
However, I’m left wondering about a few points that make it a little difficult to relate your article to my own intended usage. In short, just what are you organizing? That is, how many files are in your documents folder? and what sort of files are they? It seems like these two points are really key to whether tags can replace folders for any specific person.
Further, I’m interested in what your file names are like. By keeping everything in your Documents folder, it seems like you’d have a much greater issue with names clashing, which suggests long file names as a solution. That could lead to, in effect, sloppily encoding a folder structure into the file names. Have you noticed anything like that?
#29 Michael Barber says:@ Lucas
This script does what you’re asking for:
on adding folder items to this_folder after receiving these_items
repeat with i from 1 to number of items in these_items
set this_item to item i of these_items
tell application “Finder” to open information window of this_item
end repeat
end adding folder items to
#30 Lucas says:Michael -
Thanks for the help there. This makes tagging files a little faster until Apple adds the ability to do it right in Save dialogs.
#31 Jago says:You can also use Automator to setup a Folder Action on particular folders that either presents you with a box to type your tags in, or automatically tags any file added to folder with a preset tag…
#32 JayJay says:Just for info, you can use my application - Spotlaser - to do boolean searches and also find files starting with the @ symbol or other punctuation. It gives a bit more power to the user when you want to use spotlight. Shameless plug but if you visit the Spotlaser page, it’ll give you more info on what it can do to unleash the power of spoltlight’s architecture.
#33 JayJay says:Re the above comment: The web page is at http://members.optusnet.com.au/frovil/homepage.html
#34 Obskur3 says:I’ve been going on a recent organization binge (my goal is to severely cut my mouse usage), like Quicksilver has allowed me to eradicate my dock, so this idea intrigued me. I’m getting my feet wet by organizing with folder trees, but having a “meta” folder for files that fall into multiple tags, so I can find anything in spotlight, but also keep everything looking neat and pretty. I know it kind of defeats the purpose.
I also grabbed a demo of Hazel. I haven’t had an chance to try everything, but it looks like it has some potential. http://www.noodlesoft.com/hazel.html
#35 Kirk says:I really want a note taking application that allow the rapid assignment of multiple tags to a given note - say, 5 different groups of tags, with N number of tags in each group.
AND, more importantly, the ability to use those tags quickly. iPhoto’s approach of clicking on 2 or more tags in the keyword list instantly segregates your photos by the criteria, and can be either AND or OR (set in preferences, not so intuitive).
I want this functionality in collecting notes. I juggle many tasks concurrently, on multiple relatede topics, and a hierarchical filing system just does not cut it, as I end up needing the information filed under multiple topics, and even conditional joins of those topics.
Notae is the only note taking app I have found that uses tags, but entry is good, and usage is poor. I have even tried Supernotecard, Avenir, and Scrivner, all writing apps that support keyword tagging, and they get closer, but are not optimized to managing action items… close but no cigar.
All the GTD stuff is seriously process constrained, and non-tag friendly, except on a single tag approach - not any better than a hierarchy.
I have run dozens of applications looking for this, and can’t believe that nothing out there is available to do this, even with all the power of OS X Spotlight (no, I don’t want seperate files for each note, and use the file system as a database - arrghh!)
Suggestions? kirk@kpj2.com
#36 Noam says:Has anyone found a way to use file tagging on a Mac Server? I’m trying to develop a tagging system for our office. I know Jaguar for servers will have spotlight, but I’m not sure we want to upgrade just for that (or wait).
#37 JJ says:what does ‘GTD’ stand for?
#38 Old Shatterhand says:Problem with tagging using QS is that the File Tagging Module causes QS to bloat on CPU. On my system (with Mac OS 10.4.10) that is, to an extent that I have to relaunch QS at every couple of days or so.
#39 DLoranz says:I was wondering why tags needed an identifying prefix? - Thanks
#40 Old Shatterhand says:Tags don’t really need a prefix. But if you use Spotlight to look for e.g. ‘&mail’, it won’t return all the instances with ‘mail’ in it.
Leave a reply