Tech Erases info on $38 Bn. Oil Fund

[P8riot]

Tuesday, March 20, 2007
Associated Press

JUNEAU, Alaska — Perhaps you know that sinking feeling when a single keystroke accidentally destroy hours of work. Now imagine wiping out a disc drive containing an account worth $38 billion.

A computer technician at the Alaska Department of Revenue deleted applicant information for an oil-funded sales account — one of state residents' biggest perks.

While reformatting the disk drive during a routine maintenance check, the technician mistakenly reformatted the backup drive as well and, suddenly, all the data disappeared.

A third line of defense — backup tapes that are updated nightly — were unreadable.

"Nobody panicked, but we instantly went into planning for the worst-case scenario," said Permanent Fund Dividend Division Director Amy Skow, about the computer foul-up in July that ended up costing the department more than $200,000.

Nine months worth of information concerning the yearly payout from the Alaska Permanent Fund was gone: some 800,000 electronic images that had been painstakingly scanned into the system months earlier, the 2006 paper applications that people had either mailed in or filed over the counter, and supporting documentation such as birth certificates and proof of residence.

The only backup was the paperwork itself — stored in more than 300 cardboard boxes.

"We had to bring that paper back to the scanning room, and send it through again, and quality control it, and then you have to have a way to link that paper to that person's file," said Skow.

Staff working overtime and weekends re-entered the lost data into the system by the end of August.

Last October and November, they met their obligation to the public and a majority of the estimated 600,000 payments for last year's $1,106.96 individual dividends went out on schedule.

Former Revenue Commissioner Bill Corbus said no one was blamed in the incident.

"Everybody felt very bad about it and we all learned a lesson. There was no witch hunt," said Corbus.

According to department staff, they now have a proven and regularly tested backup and restore procedure.

[Ferris Bueller]

format C:\ ftw.

[Apophis]

That was a crazy story. Primary drive erased, backup drive erased and tape backups dysfunctional. Geh. At least no one got in trouble (yet) over it.

I'm not surprised their tape backups don't work. That seems to be a common problem in a lot of companies I do work in. Rarely do IT departments actually stick to proper tape rotation and life-cycle policies.

[jonan_]

I wonder if that tech still has his job.

[Apophis]

Originally Posted by jonan_

I wonder if that tech still has his job.

"Everybody felt very bad about it and we all learned a lesson. There was no witch hunt," said Corbus.

Doesn't look like anyone lost their job over it.

[jonan_]

oops, must have over looked that. thanks

[Apophis]

Originally Posted by jonan_

oops, must have over looked that. thanks

I wonder if he'll end up losing his job once they get all the data re-entered into the system. Maybe they just need to keep him around until they recover?

[TheFeniX]

If I never had to work on a tape drive again, it would be too soon. I have never had so many problems than with BackupExec and tape drives. Even the tech support is atrocious. The whole "library" style system is just designed poorly to begin with. We've gotten in so much trouble in the past because tape backups were telling us they completed properly with no errors, yet data was still being corrupted. And when you run backups for numerous sites, it's hard to get your boss to sign off on restore tests.

I only got to install one before I quit my old job, but the Iomega Rev Drive is aces.

I had zero issues installing it and Iomega has some top-notch guys on staff to help with an error we were getting when we first started the backup rotation. That and the price has been dropping at about e10000 times faster than tape ever would.

All that said: I never format/delete anything without making sure it's not needed. I'd rather spend an extra hour backing everything up to my external HDD than risk a data loss. That thinking has saved my sorry hide more than once.

[War.mongeR1]

Talk about a bad day at work. I feel pretty bad for the guy. We've all made mistakes before. Just not 38 billion dollar ones.

[Trooper]

Originally Posted by Apophis

I wonder if he'll end up losing his job once they get all the data re-entered into the system. Maybe they just need to keep him around until they recover?

Im sure he kept is job, now he works for the janitorial department

[P8riot]

Originally Posted by TheFeniX

I'd rather spend an extra hour backing everything up to my external HDD than risk a data loss. That thinking has saved my sorry hide more than once.

Yeah, more times that not, I come off like the hero for having that 1 extra back up right at the critical time to save someones arse.

Could you imagine that tech's heart rate at the realization at what they'd done? WOW!

[Apophis]

True story:

Many years ago back before time began (well, in 1996 anyway) I started an ISP based out of Boston, Mass. In those days it cost close to $5,000 a month for a T1 circuit and dialup was all the rage. The ISP I started was funded by a publishing company that I was the production manager for. One of the owners saw me browsing the web (yes, it existed in 1996) on my OS/2 box and became really interested in what I was doing. They decided to fund the project and away we went.

Considering the owners were not technically savvy, they didn't understand the need for a backup system. We had no tape backup and no spare drives to image the servers. All of our dialup customers signed up via telephone and we had these new account forms for each customer.

After several months of fighting for a backup system and being told "Why do we need to spend $1000 on that? We haven't had any problems!" we had a major system failure. We only had a couple servers and unfortunately the primary RADIUS, HTTP, and POP3/SMTP server was all on one box and that was the one to fail. Not having any backups I was left with a very troubling situation. I managed to go around to every workstation in the office and copied the contents of their browser cache over to my workstation and sifted through the contents and managed to recreate the entire website based on cached data from the workstations. I then had to call up a couple friends and we spent the entire night manually re-entering every single dialup account we had for both our Mass POP as well as our Rhode Island POP. This was after I spent several hours re-building the entire server (we were running a Slackware based InfoMagic distribution of Linux at the time).

It was quite a nightmare. At the end of all of this they decided that backups were important. Rather than giving me money to purchase an external SCSI drive or tape backup to archive our data, they pulled a 1.7GB external SCSI drive from the production facility that THEY used for backups and had it sent to Boston for us to use as a backup system. Production was left without a backup system at that point.

I ended up leaving the company VERY shortly thereafter.

I'm not sure why I'm telling this story, it just seemed somewhat relevant.

[jonan_]

Originally Posted by Apophis

True story:

Many years ago back before time began (well, in 1996 anyway) I started an ISP based out of Boston, Mass. In those days it cost close to $5,000 a month for a T1 circuit and dialup was all the rage. The ISP I started was funded by a publishing company that I was the production manager for. One of the owners saw me browsing the web (yes, it existed in 1996) on my OS/2 box and became really interested in what I was doing. They decided to fund the project and away we went.

Considering the owners were not technically savvy, they didn't understand the need for a backup system. We had no tape backup and no spare drives to image the servers. All of our dialup customers signed up via telephone and we had these new account forms for each customer.

After several months of fighting for a backup system and being told "Why do we need to spend $1000 on that? We haven't had any problems!" we had a major system failure. We only had a couple servers and unfortunately the primary RADIUS, HTTP, and POP3/SMTP server was all on one box and that was the one to fail. Not having any backups I was left with a very troubling situation. I managed to go around to every workstation in the office and copied the contents of their browser cache over to my workstation and sifted through the contents and managed to recreate the entire website based on cached data from the workstations. I then had to call up a couple friends and we spent the entire night manually re-entering every single dialup account we had for both our Mass POP as well as our Rhode Island POP. This was after I spent several hours re-building the entire server (we were running a Slackware based InfoMagic distribution of Linux at the time).

It was quite a nightmare. At the end of all of this they decided that backups were important. Rather than giving me money to purchase an external SCSI drive or tape backup to archive our data, they pulled a 1.7GB external SCSI drive from the production facility that THEY used for backups and had it sent to Boston for us to use as a backup system. Production was left without a backup system at that point.

I ended up leaving the company VERY shortly thereafter.

I'm not sure why I'm telling this story, it just seemed somewhat relevant.

Is that company still around?

[Apophis]

Originally Posted by jonan_

Is that company still around?

Nope. Considering that I ran all of operations within the company, I gave them a lengthy 2 months notice and detailed the issues that I had with their support of this new undertaking. They asked if they resolved the issues if I would stay and I told them if they could pull it off I would not leave, but my notice was still in effect and I would be leaving if they failed to produce.

Not that I was shocked, but they didn't actually resolve any of the issues and a few days before I was to leave they saw me packing up my desk and asked what I was doing. When I explained to them that I was leaving they went into shock and had no idea what to do. They hastily hired and I hastily trained a replacement in a day and a half. A few weeks later they had a technology issue and went off-line and IIRC it took them weeks to come back on-line. They never quite recovered and closed up shop very soon thereafter.

[P8riot]

Originally Posted by Apophis

True story:

Holy COW! WOW!