quuxpad

July 26, 2008: Windows Memory Management

Unless you’ve studied it a bit, you probably have a few wrong impressions about how Windows memory management works, and that may be hurting your understanding of how to get good performance from the OS. So here are a few links that are worth reading. On this topic, there seems to be a lot of just plain wrong advice out there on the web. So here are a few links which avoid those common misconceptions:

• Understanding Virtual Memory by Perris Calderon. This will really help form your base understanding of how the backing store works, and why several ‘common sense’ virtual memory tweaks are in fact kind of disastrous.
• The above article doesn’t cover improvements to the VMM (virtual memory manager) which are now implemented in Vista/W2003. Mark Russinovich explains the new stuff here.
• ArsTechnica has a very clued-in member who goes by the name DriverGuru. From time to time he answers questions about memory and pagefile usage. Those threads can walk you through some realworld examples, and so they help drive the message home.
• MSDN has a large section on memory management.
• The best reference is a book: Windows Internals 4th Edition, by Mark Russinovich and David Solomon. The 5th Edition is due out soon.
• Update: Mr. Russinovich has a new blog article out, which really helps to provide a more depth-ful answer to the question: Dude, Where’s My 4 Gigabytes of RAM?

The above picture is of Tmurgent’s ATM tool, very handy on Vista/W2008 (though it also works on earlier versions of the OS). I added a bit of text to the above screenshot of the tool running on my own system (2GB of RAM installed).

OK, you’re saying, that’s fine and dandy, but you don’t want to deal with a lot of mumbo jumbo. You just want to know if your system is using memory optimally! OK, here’s how to check. Open Perfmon (it’s part of Windows), remove all the default counters it is graphing, and add this one counter: Memory, Page Reads/sec. You’ll get a graph like the one below:

Your goal is to make that line as flat and low as possible while you are using your system normally. That’s the number of times Windows has to fetch data out of the pagefile on disk and stick it back into RAM. You’ll probably never get it to the point where it stays perfectly flat at zero, but you do want it to return to zero much of the time, with only occasional  spikes, and those spikes should usually be to 10 or less. Note that in this example, I set the multiplier scale to 10, so that spike to ‘60’ on the graph was really only a value of 6 page reads per second. This is a system that’s responsive and very rarely goes to disk to retreive data that should be in RAM.

If you’re seeing too many page reads, you have two basic choices: 1) buy more RAM, or  2) run less memory intensive processes. Choice #2 can mean a lot of things. It may mean just closing programs you are not using, or it may mean looking around for one or two programs which seem to hog a lot of RAM.

Tweaking your page file isn’t going to do anything to make this graph look better, and in many cases will make it a lot worse. Unless you really really understand this stuff and are in fairly unusual circumstances, tweaking the pagefile or other OS parameters is not going to be a winning strategy. Don’t say I didn’t warn you!

July 23, 2008: Exchange Admins, read this.

Running Exchange 2003? Exchange 2003 Disaster Recovery Operations Guide

Exchange 2007? Disaster Recovery and perhaps High Availability

Sometime in your Exchange admin career, you’ll be asked to recover lost data. This isn’t because Exchange itself is intrinsically a fragile platform. But if you do not set it up with recovery situations in mind, there’s a great likelihood that when one does come around, you may find yourself filled with frustration and rage.

The key here is to think of these DR procedures as proactive measures; not something you postpone until the day an issue arises. Being proactive will make that day much less likely to occur, and make the (now much less likely) disaster response a much more pleasant event.

I estimate that (depending on your Exchange setup complexity and size) the task of sitting down with this paper and bringing your Exchange to disaster readiness is a 2-3 day task. It’ll save you far more than that in the long run, but you’ll never really know. It’s like tying your shoes: how many broken legs has that saved you from?

July 21, 2008: Vista --> XP downgrade notes

If you have a legit copy of Vista Business, Enterprise, or Ultimate, you can downgrade to XP Professional. Here’s the straight dope:

• Can downgrade the Vista Business, Enterprise, or Ultimate versions only. There are no downgrade rights from other Vista versions. This is true regardless of whether your Vista license is Retail, OEM, or via Select/Open/SA agreements with Microsoft or reseller.
• OEM copies of Vista Business and Enterprise can be downgraded to XP Professional (32 or 64 bit editions) or to XP Tablet Edition.
• Vista Enterprise (available only via Select/Open/SA agreements) can be downgraded to: Windows XP Professional, Windows 2000 Professional, Windows NT® 4.0, Windows NT 3.51, Windows 98, or Windows 95.
• Microsoft does not provide a disk containing the dowgrade OS, unless you already have access to such via MSDN or Technet. Your OEM vendor may provide such a disk. There’s no guarantee they will do so for free; they may charge you for it.
• No special downgrade disk is needed; you can use any XP install disk you already have. Can be OEM, Retail, System Recovery Disk, etc.

The rollback procedure is:

1. Install XP (or other rollback OS version listed above), using its own product key if it calls for one.
2. When it comes time to activate, choose the activate by phone method.
3. Explain to the support rep that you are downgrading from Vista, and give them the Vista product key.

References:

Microsoft downgrade rights chart (Word format).

How to downgrade from Vista to XP (PC Advisor)

Vista Downgrade Rights (PDF)

July 20, 2008: Windows Scripting and CLI Reference Gold

I consider scripting to be an essential skill for a serious sysadmin. Anything you’ll have to do more than twenty times in a row is likely to lead to repetition fatigue: one or more times you’ll make some little mistake, do things a slightly different way. Scripting means spending a bit more time at the beginning, but guarantees that the results are always predictably the same. That’s a good thing even when your script screws up, because you know the same mistake was made all 20 times, and that’s a lot easier to find and correct than a random human error which was made twice within those 20 iterations.

If you’re scripting in Windows with WSH, VBscript, Jscript, or PowerShell, here are the core web references you’ll need. Googling can actually slow you down or give you bad/wrong information. These are the canonical references and they are always correct and complete. Learn to navigate these, and you’ll become a much better scripter in short order.

• Windows Scripting Reference. This is the heart of the VBscript/Jscript/WSH stuff. The part I go back to over and over again. Better than having a book at your elbow, since you can cut and paste instantly. Pay close attention to the table of contents (TOC) at left side. Illustration below highlights the most valuable bits.
• WMI and PowerShell References. These are faster to get to from this link. Again, core stuff. And again, note the TOC at left. This is how to quickly get to the bits you need.
• CLI reference. Here I chose ss64.com - quick and concise. Not fully comprehensive, but still a great reference point.
• Windows 2000 Scripting Guide. A whole book online, starting with primer guides and moving to more advanced scenarios and examples of WSH, VBScript, WMI, and ADSI (Active Directory) scripting. Don’t let the Windows 2000 in the title fool you. All of this stuff will work on Windows 2000 and above.
• Script Center. A series of articles and references for a scripter at any level from beginner to guru.
• Script Repository. This is part of the above Script Center, but deserves its own link. Here you’ll find a well-indexed set of examples suitable to quite a few sysadmin tasks.

Some folks prefer books. I’ve spent a lot of money on scripting books, but there are four books I refer to over and over and over again, while the others mainly collect dust. Here are those books and their Amazon links:

• Windows Script Host by Tim Hill. Excellent tutorial and reference to WSH and VBscript.
• VBScript In a Nutshell by Matt Childs, Paul Lomax, and Ron Petrushka. Where Tim Hill’s book takes a wider ‘learn the basics’ stance on VBscript, this O’Reilly nutshell book drills in with laser focus onto the specifics of every statement, object, method, and so on. The ‘tips and gotchas’ notes for each item are helpful. However, much of what you’ll find here is a rehash of the info found on websites linked above. A great and helpful rehash to be sure! But I go to the above web links first, and dig into this book only if I am still confused.
• Windows NT Shell Scripting by Tim Hill. There quite simply is no better book on native CMD scripting for Windows. This book explains syntactical issues that might have driven you nuts. It turns chaos into order. It is the single book I refer to most often, because cmd works everywhere, and for me, is the quickest way to a finished script, 80% of the time.
• Windows PowerShell In Action by Bruce Payette. I have to admit that I haven’t really given PowerShell as much attention as it deserves - yet. My excuse is that until it is natively installed on all the systems I will visit, CMD and WSH scripts are more instantly valuable. But PowerShell’s time is coming, and this looks to be the core book for understanding it.

And there you have it. Within these topics, I probably have 50-60 pounds of books on my shelves. Several hundred dollars, several feet of linear bookshelf space expended, and with the above exceptions, it all goes pretty much unused. With these six links and (optionally!) these four books, you can become the goto scripting guy/gal for your company.

July 19, 2008: Poor Man's Tripwire for Windows

I’ve been thinking lately: one of the more frustrating experiences in sysadmin is when you find some configuration item changed on a server, and you wish you’d known about it when it happened.

Imagine, for instance, you log on to SERVER17 and by happenstance you check the members of the local Administrators group, finding 27 accounts listed there. But you remember setting the server up a year ago with only 2 people in that group. So when did it change, and who did it, and why did they do that? But alas, your audit logs don’t go back that far, so you’ll never know. All you can do is clean it up and keep a closer watch on the server in the future.

We have monitoring solutions for uptime and performance. We have audit logs, but few people proactively monitor these (they are very verbose!). We have security auditing tools like MBSA. But these aren’t what I am thinking of. There is Tripwire for Windows, which is what I am thinking of, but it’s not inexpensive (a 2005 price quote has it at $4000 to get started, and more per monitored device).

So I set out to roll my own. And here is a first beta, which will monitor the items listed below. The idea here is that you run it via a scheduled task at regular intervals (perhaps daily or hourly), and it checks the current configuration of the server against the configuration as it was during the prior check. So when someone adds a local Administrator, or changes a service from Auto-start to Disabled, it can notify, say, a lead administrator of those changes.

•  OSVersionServicePack ‘OS Name, major service pack version, minor service pack version
•  GetLocalAdmins ‘list members of local Administrators group
•  AdministratorLastPasswordChange ‘finds date when Administrator account last changed password TODO: is account disabled?
•  ListServices ‘list all installed services and their startmode
•  ListSoftware ‘list software installed (via Windows Installer)
•  ListHardware ‘list installed hardware that has a PCI vendor code
•  ListDisks ‘list disks (except network drives)
•  ListRAM ‘list each RAM stick, size, and status (if available)
•  ListNetwork ‘list each IP-enabled NIC and the first IP/Gateway bound to it
•  ListScheduledTasks ‘TODO list Scheduled tasks with status and minimal schedule
•  EventLogCriticalErrors ‘TODO return critical errors since last run (maybe this is out of scope)
•  WinsatPerfTests ‘TODO if Vista/2008, run winsat perf (maybe this is out of scope)
•  OpenPorts ‘TODO ports listening on the server
•  NTRights ‘TODO NT Rights assignments
•  GPOs ‘TODO domain and local policies
•  BootIni ‘TODO Boot.ini
•  LocalUsers ‘TODO all local users (on nonDCs!)
•  Reboot ‘TODO notify if system was rebooted. Can we notify *before* system gets rebooted?
•  EnvironmentVariables ‘TODO track system level env vars
•  Routes ‘TODO route print info
•  FirewallConfig ‘TODO firewall configuration
•  TimeService ‘TODO time service config
•  HostsFile ‘TODO check Hosts file
•  LmHostsFile ‘TODO check LmHosts file
•  AuditConfig ‘TODO check auditing configuration
•  BackupOccurred ‘TODO see if backups are happening
•  EventLogConfig ‘TODO check Event Logging configuration
•  DomainRoles ‘Determines domain membership, if system is domain controller, and if yes, some DC properties
•  FSMOroles ‘if DC, list FSMO roles

Right now the script has no way to send mail or SMS or whatever; I’ll be adding that later. I’d love to get your feedback on this thing. Again, you can download it here.

July 15, 2008: Writing better

A sysadmin often needs to write convincingly and well. Emails to the boss, the users, other sysadmins. Upgrade justifications. And so on.

This short essay, written for college students in the 50’s, is full of timeless good advice which can help your writing become more immediately useful to its intended audience.

July 14, 2008: FSMO roles command.

Quick and dirty. You want to know which domain controllers hold the FSMO roles in your Active Directory domain? Hate navigating through all the dialog boxes in the MMC tools and find NTDSUTIL navigation a pain?

On your DC, drop to a cmd session and run this - all as one line:

Ntdsutil Roles Connections “Connect to server %COMPUTERNAME%” Quit “Select Operation Target” “List roles for connected server” Quit Quit Quit

It will list the roles lickety-split, and you’re done!

July 13, 2008: winsat perf-testing tool

I’ve been trying to get a CF card to work as ReadyBoost memory. Still no success on that project yet; I may have more to say about that in the future.

But along the way I have discovered a neat little command line tool called winsat. This is basically the commandline underpinnings of the Windows Experience Index tool, except that it returns real numbers, rather than something fuzzy like “2.1, based on lowest subsystem score.” It was James O’Neill’s blog which tipped me off to this nifty little thing.

Let’s take it out for a quick spin. Here I am testing random IO performance for reads from the CF card (which is drive F) for large chunks of data (524288 bytes to be exact):

Overall, the disk is capable of 14.42 megabytes per second on large random reads. For comparison, my main C: drive gets 59.75 on the same test and the GUI Windows Experience rating is 5.3. So this CF disk is no speed demon, but should meet the ReadyBoost criteria.

Winsat can also test CPU, memory, and a few other things. Here are the full docs to the commandline tool. You can also access the winsat API via scripting methods, and via more formal .NET coding methods. Given that it’s built in to Vista and Windows 2008, this should be a handy tool in your troubleshooting and system baselining arsenal!

OK. So now I’ll go back to trying to make ReadyBoost work on this CF card. Wish me luck.