Time May Change Me, But I Can't Trace Time

The title of this entry is from the David Bowie song "Changes" off the album "Hunky Dory". While I know he didn't have computer data in mind when he sang these words, it sure applies. How do we trace time? Can we track changes in data and adjust resources based on change? The answer lies in another quote from Maestro Bowie "It Ain't Easy".

I've been researching a paper around the idea of just-in-time data protection. It pulls together some concepts that I have been batting around for quite some time including the idea of Service-Oriented Architectures and dynamic data protection. I've written about both of these topics before. In looking at how to make the data protection environment more responsive, I started to realize that resource allocation needs to be adjusted according to how quickly the data is changing. The rate of change of the data, should then drive one of the major data protection metrics, Recovery Point Objective or RPO. RPO basically says how far back in time you are committed to recover some piece of data. My thinking goes like this: Why spend money to provide high performance backup to data that isn't changing? Conversely, rapidly changing data justifies a short time frame RPO and more resources.

As I went about talking with vendors and IT people I quickly discovered that there was no good and easy way to determine the rate of change of data. We simply don't track data that way. There are some indirect measures, especially the rate of growth of disk storage. For homogeneous data stores, such as a single database on a disk, this works well, assuming your unit of measure is the entire database. It doesn't work well at all for unstructured data, especially file data. We might be able to look at how much space is used in a volume but that's a pretty gross measure. Databases have some tools to show how fast data is changing but that does not translate to the disk block level and does nothing for file systems.

What we need to understand is how often individual files are changing and then adjust their data protection accordingly. If a file is changing an awful lot, then it might justify a very short RPO. If it's not changing at all, perhaps we don't need to back it up at all so long as a version exists in an archive. In other words, we need to assign resources that match the metrics and rate of change affects the metrics. This is complicated because how often data changes is variable. It might follow along with a predictable lifecycle but then again, it might be more variable than that. The only way to know is to actually measure the rate of change of data, especially file data.

The simple solution is a system that tracks changes in the file system and calculates rate of change for individual files. This information would then be used to calculate an appropriate RPO and assign data protection resources that meet the metrics. The best system would so this on the fly and dynamically reassign data protection resources. A system like this would be cost effective while providing high levels of data protection.

No one has this yet. I'm thinking it is two to five years before this idea really takes hold in products. That's okay. It's usually a long time before something goes from a good idea to a usable product. We have to start with the good idea.