travis' brain dump

Yes, for those of you who are not interested, read no further. LOL

M.O.M. or Microsoft Operations Manager can be a useful and valuable tool… when used correctly. But what good is an installation of MOM if it’s not doing anything?

With the smoke still clearing from the storm I caused at the last place I was at in followup to my post about taking pride in your job, I have found myself looking down the barrell of a similar gun. What do these people do all day? I mean seriously.

I come into this place with a few tasks… work on the event’s management software and MOM. Those are my two tasks. What do I find? No mails being sent from any of the monitoring software because the McAfee policy is blocking the executables from sending out mails to tell you there are problems out on your network and last but not least the MOM database server is FULL!!!!! (not to mention it is full of alerts from almost over 700 days ago too…) *sigh*

Seriously folks… Why is it people like myself have such a hard time getting jobs when we’re looking (thankfully not the case with this job I’m at right now) and yet people in these jobs are worthless. I’m not going to judge the person I’m filling in for… It appears the guy has way too much to do but allowing your monitoring server to go down in flames isn’t exactly keeping on top of your job. *shrug* Maybe it’s not on his priority list… maybe he’s a slacker… maybe there really is just too much going on for him and he can’t keep up with it.

Whatever the case may be, this is two for two people. Twice I’ve been put in front of someone’s network and twice I’ve found it’s neglected and screwed up. At least it’s something productive to do while I’m working on some permanent work. 🙂

I need to get back to the over abundance of shrinking of databases that need done here, just figured I’d rant.

late

So I get a phone call this morning from a friend of mine at a landscaping company… evidently during the move of their primary server something went awry… The system no longer wanted to boot up and was complaining profusely. I made time as quickly as I could to come down to fix them up and much to my surprise when I arrived found that the worst case scenario was staring me in the face.

The system had been rebooted during bootup… One of the drives in the RAID array was showing as missing/failed and the state of the entire RAID volume was being listed as ‘ failed ‘ … uh oh … I would have happily taken a sign that the volume was running in a diminished capacity but this thing was ANGRY. Somehow by all grace I was able to bring the volume up into a rebuild state, but it was still unhappy, so I installed another disk to boot from and load up some utilities. The array went about its rebuilding just as happy as could be… After taking an hour or two to do so, I was able to then try to run CheckDisk on the partitions. The OS partition went pretty well, but the Data partition was all jacked up…. time to call in data recovery software… *fingers crossed* Currently it’s doing a sector scan of the drive and is at 38%… I’ve been at this 6 hours now. I’m hoping for some good news after this and hopefully be able to restore the financial files at the very least. Anything else on top of those is just icing on the cake.

I tell ya. This kinda stuff scares me. I’ve been put in this position two other times and it was uncomfortable then as it is now. It always sucks when you’ve got to do your best to restore what you can but in the end, regardless you have to hand them a pretty hefty bill no matter how much data you recovered. *shudder* I guess this little instance will nudge them towards some better practices in data backup. A positive in all of this. Let’s just hope it’s not at the expense of losing financial data. 🙁

43% – It’s picking up some speed. Well here’s hopin for a good report at the end of this… I’ll post up more later as I think I’m going to be at this a while. After I recover data, I have to reinstall the entire server. Then it’s rejoin the domain time for all the workstations which happen to be offsite so that’s going to have to wait until next week I think. (not sure when they are moving the rest of the workstations over here)

Cross your fingers for these guys folks, they’re gonna need it.

ahhh yes… this was the scene this morning in Breckenridge. With only 53 days until the official opening of the season there, the fever is kicking in… No official word yet from ABasin as to an opening date, however if trends speak anything it will be a couple weeks before Breck and Keystone open up for sure. Last year was October 13th… I can’t wait. 🙂

Probably one of the most hashed out battles of all time concerning backups and data retention. Brick-Level Backups or individual mailbox backups, while being an eye catching feature, are evil on oh-so-many levels. At first glance they look like mana from heaven, but in all actuality it’s not. While popular backup software tends to tout this feature and promote it as an easy method for backing up and restoring individual mailboxes as opposed to the more bulky and rather cumbersome Information Store backup it’s all something that should be ignored. PERIOD!There are instances where some people truly understand the issues at hand when it comes down to brick-level backups and do actually make some use of them. Hats off to them for living with the problems. Seriously folks, if it was just an issue of this thing being a beast, I could live with it.

The biggest problem with brick-level backups is that it doesn’t do nearly what it says it does. On paper an in presentation it looks great. Backup your users individual folders and you’re good to go for easy restores should one of them do some damage and delete the wrong thing. In the current position I’m working at (and what has spawned this blog/rant) they are running into the same problems just about everyone runs into: Corrupt Items or Access Denied (usually on those corrupt items). The solution for most people (seeing as how this solution comes from the makers of this backup software) seems to be removing the warning messages about the backup of said corrupted messages. While that gets rid of the messages, it also removes the warning system for potentially valid error messages and could cause your backup to be as useless as a screen door on a submarine. Also the false notion that a brick-level backup is sufficient for disaster recovery is a common problem. Most of the points against brick-level backups out there are directly related to this. A disaster is just that… total loss of data, not Joe Bob’s missing mail from his girlfriend who wants to meet him for lunch or an accidently deleted meeting request. There are functions in place for that provided by Microsoft. USE THEM!

While the argument stands that you have to do an entire IS restore when you backup on the the IS, I think it’s important to take a look at the problems you’re going to face and have to fix when dealing with Exchange Administration.

1. Joe/Jane User accidently deleted an important email from the CEO
2. Joe/Jane User jacked up some mail items and corrupted the crap out of them so bad they can’t access that item again
3. The Admin smoked some crack this morning and deleted a mailbox

Well, problem #1 on the list there is pretty easily handled by a little nifty item called ‘Deletion Settings’ located in the Limits panel under the mailbox store properties on your Exchange server. You can set how long the server will actually hold on to the items that have been “Permanently Deleted” from your Deleted Items folder. (scary huh?) This actually allows users to go back and restore individual deleted items on their own without having to ring your desk and make you do a complete restore operation. (which by the way is a complete waste of time for just one single mail item)

The flipside to that is problem #2… This is kinda a mix since Deleted Items Retention  doesn’t really help. This one will require an IS restore. Painful, but unless this is a critical mail that contains some meaning of life stuff it’s something that can be more than likely retrieved by sending off a mail to the originator, making a phone call and requesting a resend or otherwise finding an alternate method of getting that information back. If it’s an email from God then more than likely your Admin will make the effort. If he/she is an athiest you’re probably screwed.

Leaving only problem #3 here… If his happens you’ve got much bigger problems than a missing mailbox. Check to make sure you’re not missing company assets to bankroll his/her drug problem. Seriously though… If an admin makes a mistake this big… they can afford the time to do an IS restore … they will need the company time to use the internet to look for a new line of work. 🙂

Here’s some of the key points and problems with using a brick-level backup.

1. MICROSOFT SAYS NO! If you’ve even taken half a second to read any of the documentation or you’re certified in Exchange you know as well as anyone that the documentation strictly says to backup the database files online using IS and DS or offline in a flatfile backup. Notes around concerning brick-level are well… less than happy. 🙂 I’d say #1 is the first and foremost reason to not do it, listen to the people who wrote the software… they might actually know what’s best!
2. The interface. Brick-level backups use MAPI to connect to the mailboxes. Hello? Isn’t this the CLIENT interface for handling sending mail, retrieving mail and so on for an individual mailbox… what qualifies this to be an active method for moving LARGE amounts of data at any given point in time?  Let’s not forget that by using this method you’re increasing the size of your backup since duplicate items are stored as one item in the IS but when pulled off using MAPI on a per user basis creates individual items for each duplicate under each user identity. This I believe is also the biggest contributors to the whole corrupted item problem in some way. Let’s not forget the fact I love Veritas’s explaination about how to fix this. ‘ Just delete the user mailbox and rebuild it ‘ — umm… If the items that were corrupted won’t backup, what happens to those when you delete the mailbox and restore? KISS MY ADMIN REAR END!
3. Transaction logs are nixed out of the question when you rely directly on brick level backups. Did you know when you backup your transaction logs along with your IS you can often restore everything down to the minute you had the problems?  (also a good reason to keep your transaction logs on a different drive, lol) If you restore the IS and replay your transaction logs you can 99% of the time get everything you were missing from after the backup restored to your IS. Can’t do that with brick level! 
4. Time on the back up is saved when using IS backups. I don’t know about anyone else, but that blasted brick-level backup takes 4-5 times longer than the IS backup takes. Thursday backups here are a pain since they are a complete and non-incremental backup. They run well into Friday and that’s not cool.

I think the biggest point being made here is to use the tools provided by the makers of the software first and foremost before relying completely on third party software to do the work. Microsoft has thought this through and tried to do their best to find time and cost effective ways of handling the day to day problems while still supporting a disaster level solution. Use deletion settings in Exchange and show your users how to use the ‘ Recover Deleted Items ‘ option under Tools in Outlook. When someone leaves your company, don’t delete their mailbox immediately…. just stop the flow of mail to it and keep it sitting there waiting until you’re sure no one needs the items. If you practice diligence in planning for disasters and do things such as setting up a restore server, you can still go back and restore deleted items.