Just-in-time Data Protection

There is more activity in the data protection arena now than I've seen in years. We are reaping the fruits of years of new innovation to finally begin to achieve something special in data protection. Looking back over the past few years we have seen the emergence of:

• Disk-based backup including VTL
• Data reduction techniques
• Continuous Data Protection (CDP) and real-time replication
• WAFS or Wide Area File Systems
• Tiered Storage
• Data Lifecycle Management

All of these combined provide a whole host of choices in how one architectures data protection systems. It would seem that soon we will have the ability to protection all of our data all the time. But does that make the most sense?

Constant and complete data protection will always be very expensive. Bandwidth costs money, disk capacity costs money, and software costs money. To strive to backup everything all the time is to strive to waste money. Why? Because there is rarely a good reason to backup all the data, all the time? We may want to backup all of it sometimes. That the realm of tradition backup. We may want to backup some of it all the time. That's were CDP comes in.

Where we send it to be protected is also part of the equation. Do we keep it on-site? Copy it to tape to be stored off-site? Send it over a WAn link to another locations. All of these location options provide different levels of protecton at different costs.

What we really want is "just-in-time" data protection. Akin to JIT manufacturing, the goal is to protection data when it needs to be protected. Some data will need constant backup, perhaps because it is mission critical and needs to be available 24x7x365. Other data will only be backed up occasionally because it doesn't change much. Image files are an example of something that rarely if ever changes and only needs to be backed of once in awhile. Some data will be copied off-site to a DR facility immediately because it must be made available the second a disaster occurs. Other data will be archived and left in a vault on site. It all depends on how important the data is, how fast it needs to be accessed after a disaster or failure, and how much you are willing to pay for the priviledge of having the other two.

New data protection technology allows us to fine tune our systems to deliver data to backup exactly when it is needed. If we follow a philosophy of JIT data protection then we will soon find an equilibrium between protection, cost, and time. And that is a perfect combination. Start with the JIT goal and you have a fighting chance of achieving that balance. Do too much or pay too much and you will waste resources. Do too little or pay too little and your important data is always in danger.

Treet - Part II

When I wrote my column on treet (like spam only in blogs) I would have though that the morons who distribute this stuff would have got the message. Hello! I'm hostile to your business model. Lo and behold! They are at it again. This time it's a doozy.

The latest unwanted garbage in the comments section of my blog promises to help you earn a degree in two weeks. The "degrees" that you can "earn" run the gamet from a simple BA to a PHD. Take note, not a B.A. or Ph.D. Now, my bachelors took 4 years to accomplish and my wife's Ph.D. took something like an additional 6 years. Since I know we are not slow learners (most people with doctorates aren't) I can only assume that Ph.D. and PHD are not the same thing.

If you have the wherewithal to get an Internet connection going and bother to read through blogs, are you really that below average in intelligence? I doubt it. Who then is actually calling this number? So, I said to myself,"How about me?". I called.

The message only identified them as "The University". No other name. Like "The Village" in the old TV show "The Prisoner". I was expecting to hear "For Number 2, press 2. For Number 6, press 6. For number 1... that would be telling". If you haven't seen the show then that's too bad. Go find it. Watch it. Love it.

Anyway, all I was able to get was an answering machine. Always a bad sign. The messaged asked for two phone numbers and an e-mail. That's much more information than most answering machines ask for. Hmmm... I tried again. The message said that you could dial your party's extension at anytime and it would connect you. Liars! I tried an old voice mail trick. Usually if you dial a "0", "*" or "#" you get an operator. Nope. 0 repeats the opening message, # doesn't register, and the number 1 just cuts to the answering machine. Hmmm again... Any other number is ignored (despite the fact that some of them had to be extensions) and * asks for a PIN code. That indicates that it's a cheap answering machine not a professional PBX. In any event, you can't do anything but leave a voice mail and there's no way I'm doing that. Especially with a "university" that doesn't have a name. I don't need the guys in black boots coming for me in the night.


All the while I kept wondering, who could possible think this is for real? That got me thinking. Who actually responds to these things? Do the marketers of these phony degrees actually think that my gentle readers are this stupid and gullible? Is anyone this gullible? Phishing I get. Some of those phishing e-mails look very real. I can see how even a smart person could get caught by a phisher. But this? It simply doesn't make sense.

A quick google of the number shows that this same message has been planted like a kudzu in many other blogs, mostly technology-oriented ones. That makes it even more puzzling. Technology people tend to be educated and intelligent. Even if they needed additional degrees, they couldn't possibly be scammed by this. Once again, treet is a waste of time in ways that spam and phishing aren't. That makes it seem just that much more stupid. A traditional con man or grifter has to pick their mark carefully. If the mark is suspicious, too smart, or even too experienced, then the scam won't work. I can't imagine anyone capable of operating a computer being that inexperienced, gullible, or just plain dumb.

This simply confirms my initial hypothesis. The people that treeters are trying to scam are not the stupid ones here. It's the treeters themselves that are dumb. So, if any of you treeters are out there, contact me. I'm dying to know how you make money at this. Even Bill Gates, who seems to be able to make money at almost anything, probably would be interested. Oh, and I have a bridge in Brooklyn that you might be interested in. Real cheap. Trust me.

iSCSI at Home

After the last couple of posts, I guess I need to play this one straight. So let's talk about a way out there idea - iSCSI at Home. Okay, not really at home but for the SOHO market.

That probably sounds crazy to some people. Most SOHO applications don't need SCSI or SANs, right? Wrong! Sharing resources is as important to a small office environment as a big one. Most of the time that means files, and files mean NAS. However, there are a lot of applications where that is not the case. Let's name a few:

• Graphics and video - These folks have used SCSI-based systems for ever. These type of applications are now even common in homes thanks to Apple.
• Music - making, not listening. Many amateur musicians have small home studios.
• Data Protection - small and remote offices have data protection needs too, even if it's only tape backup
• Shared disk resources - sure you can do it with NAS but having a private disk volume all your own on a shared device is cool too.
• Multimedia - whether it's your home theatre or a company presentation system, the file stack is a real hinderance to performance .
  

Okay, so the best application here is data protection but that's important. Even home systems full of MP3's and kid's homework need to be backed up. Sadly, most are not. You can use simple freeware programs but that's not for a commercial venture. The better solution is to have something more industrial grade. That's where iSCSI comes in.

iSCSI is cheap and has the horsepower to perform most major data protection functions. The perfect data protection system for the SOHO market is block level CDP to an external drive or small array, priced at under $1000. Fire and forget. You can do this with direct attached storage such as a USB drive but that gets expensive in an office with more than one computer. And what office doesn't have more than one computer. Heck, I've got five in my home! Most offices have as many. A single iSCSI-based disk backup device with CDP running under the hood (a Windows service) would provide heavy duty (by SOHO standards) protection. Again, this can be done with NAS devices but not as efficiently. Fibre Channel is too difficult and expensive. A small iSCSI device running over conventional Ethernet (not GigE. Please! Even though it's getting cheap it's overkill for most SOHO customers) that does plug and play would be attractive. More attractive than have five separate backup devices to mess around with anyway.

Price is the problem of course. Individual single disk backup devices are under $200. Even a more sophisticated system would have to be well under $1000, more like $500, to be attractive to the SOHO market. Remote offices would pay more since they come out of big corporate budgets. This is why USB disk drive devices are so attractive. iSCSI runs over existing infrastructure and if SATA drives were used, could be made cost effective.

Think about it. And then send me one to test out/user/keep forever. I have these five machines you see....

Grinding my Axe on Buy.com

It's sad. Very sad.

When something that has worked well for so long is utterly and inexplicably ruined, one can't help but feel a sense of loss. We feel this when an old friendship disintegrates or we watch a storied brand eaten by a corporate titan (does anyone remember AMC cars? The Rambler was a classic and the Javelin and Pacer at least interesting.) Unfortunately this has happened to Buy.com. Sigh...

Buy.com started life as Buybooks.com in the late-90's. At a time when it was not certain that Amazon.com would dominate online books sales, a number of Internet book sellers popped up. Buybooks.com was one of them. What set them apart was consistently good prices, a web site that gave accurate information, and great customer service. It's a damn shame that this is no longer the case. My experience last week is a case study in how to annoy long-term, loyal customers. After ten years of being just that, I find it sad, terribly sad, to see what they have become.

My son's birthday was coming up. Now that he is a teenager, his toys are looking more like my toys. Since he kept asking about borrowing my MP3 player, I thought it would make a great present for him. As has been my practice when I wanted an electronic gadget, I went up to Buy.com, looked for the item I wanted, checked to see if it was in stock, then ordered it. I carefully figured out how long it would take (worst case) to get here. More than enough time, I said to myself. What I wanted was pretty specific too. The new SanDisk Sansa players allow you to add in SD memory cards. That makes them expandable. Since it is likely that his future music collection will be mostly electronic, the ability to store music on cards and swap them in and out seemed sensible. Think of it as futureproofing. I was definitely pleased with myself.

Ah! The sin of hubris. I should have known that disaster was lurking. As the day approached when I would present this neat little gadget to my first born, I became anxious. The present had not arrived. I check the Buy.com website and - HORRORS - it was listed as "On Order". It had not even shipped! What now? I quickly looked up "On Order" and discovered that it was a euphemism for "we don't have it and don't know when we will have it". Panic time.

I next did what one does in this situation. I went to call customer service. One hitch though. The phone number for customer service is nowhere to be found on the Buy.com site. I'm not some Internet neophyte. I know where they hide these numbers. It simply wasn't there. Well, they can run but they can't hide. I had an old invoice that had the number on it. I double checked the Internet forums (there is a forum for everything) and found the same number. I called the number. I wish I hadn't.

Back in my days in the call center industry, I helped the company I worked for establish an early overseas call center service. Talk about your past coming back to bite you. The Buy.com call center was, shall we say, culturally insensitive. I explained to them that this was a birthday present and I needed it right away. I was treated with indifference and silence. Incredible. Not even "I'm so sorry sir. We apologize for the mix up." Nope. They did not even acknowledge that this was a problem. Their attitude was completely cavalier.

You might be wondering why Buy.com could not ship something that was in stock. Turns out, even though the website says something is in stock, it doesn't mean that it actually is in stock. They only update inventory on the site once a day. I thought they were kidding. Live inventory lookup has been around since Buy.com started business! They even admitted that they had no way of knowing when an out-of-stock item was due in. Once again, I wondered if they were joking. No ETA on popular items? Not even an estimate? No way. How 1970's.

Now, typically when you have an upset customer, especially when they are upset over your screw up, you offer to make them whole. Fix the problem in some concrete way. Sorry. Not happening here. I told them what I wanted since they didn't offer it. I asked for a different unit that was in stock, and which cost the same as the original, with free overnight shipping. Instead they offered me a gift certificate for $10 off my next purchase. Like that will happen before the next ice age. I hung up.

Okay, I can't leave it at that. Not me. So I called back. I figured that if I talked to a manager I might get some satisfaction. What was I thinking? The manager was only slightly less rude. I once again said what I wanted and she claimed she couldn't. Here's where she slipped. She said she had to follow certain cost guidelines. That blew the top off my head. I asked her if she was really willing to lose the entire sale and a long-term customer for the sake of a few bucks. She corrected herself and said it was just guidelines not cost guidelines. Too late! The genie is out of the bottle. Buy.com is willing to nickel and dime their customers. I got that now. Thanks for the moment of honesty.

She offered me the same gift certificate which she said could be used to cover overnight shipping. So I had to order another product and cancel the current one. "Okay", said I, "let's do that". Now, here comes the kicker. Wait for it.... She couldn't do that either! I had to go to the website and do it myself. I was almost speechless, which is unique for me. Customer service can't take an order or cancel a product? That's not right. It's almost unbelievable. I hung up again.

I ran out to a local Circuit City and bought the same thing for ten bucks more. That's ten bucks well worth spending. The clerk was nice. When I couldn't' find what I was looking for on the shelf she went out of her way to find one in the back. Whereas Buy.com had my blood pressure in the stratosphere, this simple act of service made me feel good to be shopping.The clerk even wished my son a happy birthday. I'll go back there.

Buy.com continued to add insult to injury. I canceled my order on the website. Fine. Done with that. Not so. It turns out that they are refusing to cancel the order since it has already "entered the shipping phase". It's shipping... Except that it still hasn't arrived at their warehouse and they can't tell me when it will. It's like a twisted version of a Buddhist koan "How can something be shipping when it is not there to ship?" The email said I could return it, refuse it, or keep it (and presumably pay for it). The chances of me catching the UPS guy to refuse it is negligible and I don't want to keep it. So it's more work on my part just to get my money back from them. I will probably have to wrangle with them for months to come, all the while my money is locked in their vaults.

So what is the point of this lengthy rant? To begin with, it's quite therapeutic. But I don't post stuff like that. I don't want to waste my readers time. Instead, it is here as a warning to companies that hope to shave a few points off their expense lines. Customer service is a bad place to do it. You will lose your most loyal customers and everyone they know. What is gained this quarter in expense reductions will be paid for time and time again in lost revenue. I see so many businesses doing this. They short change customers to make a few pennies float down to the bottom line. They then wonder where all those customers have gone. As Clint Eastwood says in High Plains Drifter, "Out." Recently, someone told me they would never buy anything from Dell again because of similar lousy (outsourced) service. I won't either now that I've heard that.

So, sales and marketing people, when the accountants tell you cut back on service or outsource to someplace overseas, think about what you've read here. Say to yourself "Okay they get the expenses down but how do I make my numbers?" Customer service is that ounce of prevention that keeps customers from flying toward the competition.

Amazon.com wins while Buy.com loses. That's what happens to cheapskates.

Two Great Tastes That... Might Taste Bad

The big buzz in the data storage industry this week is around the acquisition of ADIC by Quantum. The stated goal of the merger is to create the world's largest data protection company. Okay....

Now, don't get me wrong. I like both of these companies. The folks at Quantum are real nice and know a lot about tape backup. I even mentioned them in my book on data protection. ADIC also makes some decent and innovative products. This merger, however, leaves me cold.

To begin with, if you are an ADIC shareholder, you're getting hosed. Screwed, nailed, taken to the cleaners. Last year was a profitable one for ADIC. Net income for fiscal year 2005 was $14.27M with operating income at $4.09 on $454M of revenue. Not bad really. They made money. Contrast that with Quantum. Last fiscal year they lost $3.5M. Worse, operating income was ($9.54M) on revenues of $794.15M. There's a reason that number is in red. It's a loss.

To add insult to injury, ADIC is selling out for only $770M. That's only 1.7 times revenue. Not that high a valuation for a technology company and it's an all stock deal. That is even less exciting when you consider that Quantum's stock is only about 3 bucks a share and has traded in a narrow band between about $2 a share and $4.5 per share since the middle of 2002. There does not seem to be any share growth or profits coming out of Quantum, so what are ADIC shareholders getting for their company?

From a product perspective, the merger makes even less sense. They make the same stuff! Sure Quantum makes the tape drives that go into the libraries that both sell and ADIC has some nifty file system software. However, the role of tape in data protection is diminishing. It is quickly becoming an archive medium and not for primary data protection. Disk systems are the future of data protection. Global file space software is a good thing but so heavily tied to server systems that it is hard to imagine it having much of an impact on future revenue for the combined company. The product overlap is big enough that the combined company gets very little technology edge from this merger.

Let's recap. ADIC shareholders trade shares in a profitable company for those of an unprofitable one with no share price growth. Quantum gets more of the same from a product and technology point of view. The combined company has to cut costs and go through the merger causing all kinds of disruption. Where's the value?

The only thing that comes to mind is customers. Quantum and ADIC compete for many of the same customers. Big systems and storage vendors OEM their products and the same end-users buy their gear. Is this the justification? To add ADIC customers to the Quantum fold? Eek! I hope not. ADIC customers, faced with no ADIC, are just as likely to open up future purchases to all vendors including all the new kids on the block.

The folks who will benefit the most will be IBM, SUN, Dot Hill, etc. Sun's data protection portfolio is much better now after the StorageTek acquisition (which made sense), IBM and SUN can offer complete solutions. Even in the OEM business, an important revenue line for ADIC, this is true as competitors, large and small, run in to take the ADIC business away from Quantum. If Dot Hill doesn't have a field day with this then they are trying not to succeed.

I am reminded of an old Martin Mull bit called "Sweet Steaks". It's a fake commercial for a children's cereal. Really funny. One of the "claims" made in the piece is that when Sweet Steaks are "served with fruit and milk, it has all the nutrition of... fruit and milk." Same here. When I do the math I come up with one plus one equalling zero point 5. ADIC should have waited for a better deal.