Preventing problems

You may have heard the saying, "An ounce of prevention is worth a pound of cure." This is especially true in the computing environment. In the support arena, we see many customers who could have saved enormous amounts of time, money, and energy, if they had just taken some time up front to plan. IBM computers are exquisite machines, expensive and complex. They are the muscle behind your business. To squeeze out every pound of performance, invest in an ounce of prevention.

Preventing problems Preventive maintenance Maintenance strategy Stay informed with automated alerts Version information

Preventive maintenance is the art of heading off disaster by careful planning and regular upkeep. The Support for eServer pSeries servers webpage is going to be your single best source of information about preventive maintenance. Look up the information by pSeries model. The hardware and software information for each model will have a section on prevention. This section will contain links to other IBM resources, white papers, redbooks, and anything else that pertains to the upkeep of this particular model of pSeries and operating system.

Get a copy of the service guide, handbook, or redbook for your particular pSeries machine from the Support for eServer pSeries servers webpage. Look up in the hardware section. There will be links you can follow. Or, you can look directly into the redbooks area for for your machine's handbook. Search by model number of machine.

Check the IBM eServer pSeries Information Center for AIX 5L versions 5.1 and 5.2 or pSeries and AIX Information Center for AIX 5L version 5.3 for service and user's guides to help you make informed decisions about upgrades. You can look up information by operating system or by hardware model. Most of these guides can be downloaded over the internet, read on your machine, and printed out. If it's a redbook, it can also be ordered on CDROM.

Make a point of being familiar with the contents of these service guides or handbooks. They will have information that may be hard to find elsewhere. It's not only the best reference material for your machine, it's also a good tool for planning and preventive maintenance. Sections on "RAS" will be especially relevant to preventive maintenance. RAS stands for "Reliability," "Availablity," and "Serviceability." These RAS qualities are important to, and affect, preventive maintenance. The higher the RAS, the longer your machine stays in operation, the better it performs, and the easier it is serviced. For example, the newer pSeries machines with the more advanced AIX versions have some fault recovery and automated calls for support.

Plan for periodic updates of your operating system and microcode. AIX "Recommended Maintenance Packages" and microcode fixes come out several times a year. In these fixes are code that prevent known defects. To apply these fixes is to ensure that your machine does not run into a problem that has been investigated and fixed by IBM. Should you have a computer problem, and IBM technical support notices that you have not applied any of the recent fixes, you will probably be told to upgrade. This is a debugging technique that eliminates all known problems from the mix and allows the technician to confirm that the problem your computer has is due to something new rather than something that IBM has already fixed. Wouldn't you rather apply fixes to your machine regularly, when you can schedule convenient down-time, rather than be asked by support personnel to upgrade during an emergency situation?

Here are some general guidelines that can become the basis for your personal maintenance strategy. Take these ideas and modify them if necessary, adjusting them to your particular computing environment.

Upgrading

It is advisable to upgrade your operating or your hardware upon these conditions:

• You've received a "cert advisory"
• You've received an e-mail from the Subscription Services for UNIX Severs saying that a security fix has been released.
• You've received an e-mail from the Subscription Services for UNIX Servers saying that a Recommended Maintenance Package has been released.
• AIX or pSeries technical support recommends that you the upgrade the operating system or microcode.

Thorough testing

When upgrading, please keep these ideals in mind:

1. Fixes and hardware upgrades should be applied in a testing, development, or staging environment. Testing should occur on a non-production machine before moving the fixes or hardware to a production environment. If you don't have a testing environment, you may be able to arrange one through the IBM Testing Services National Test Center.
2. Make an operating system backup ( mksysb) before applying fixes. Backup any application data or databases before applying fixes.
3. Maintenance Package upgrades and microcode upgrades should be done separately rather than at the same time. If you apply both types of fixes at the same time, and you experience a problem, you will not know which group of fixes caused the problem- the microcode or the Maintenance Package. Apply groups of fixes one at a time and monitor the changes. If you need to apply both microcode fixes and a Maintenance Package, apply one and watch it for a week, then apply the other and watch them both for a week. If all is well, proceed with 30-60 days of testing. If all is not well, back off the last one you applied and see if the problem goes away. This will help you distinguish between the two, should a problem be introduced by one of the upgrades.
4. The testing of fixes should be done for 30-60 days:
   • in the testing environment
   • with all applications that the production system has
   • simulating a stress load that is a least as high as the production environment if not more
   • ensuring that you have the latest copy of base media available before attempting an upgrade
   • applying operating system fixes using the "alt_disk_install" technique
   • applying operating system fixes (rather than committing them) so that they can be backed-off easily should an emergency occur
5. Once the fixes pass inspection in the test environment, apply them in the production environment, making sure to have a large enough maintenance window to apply the fixes (including some time to back off the fixes if necessary). Only commit the fixes in production after a long period of monitoring which includes a time of high stress and load.

Anticipating growth

Every business should anticipate their growth. It is important to project the growth of the stress on your production machine, and test your hardware, operating system, and application software knowing what the likely stress and load on the production system will be in the future. This is called "capacity planning." A machine operating at its maximum capacity in terms of use of memory, cpu, and i/o, has no room for growth. Should your company experience a sharp increase in end-customers demanding your excellent product, the load may exceed the limits of the production system which was previously operating just fine at maximum capacity.

IBM actually prefers that you test above anticipated stress and load levels so you can tell whether or not your hardware, operating system, and application software can accomodate a large spike in load and stress. If a production machine is operating at maximum capacity at all times, you need to look at either purchasing an additional machine to share the load, or upgrading your hardware to something that has more memory, cpu, and i/o capacity. Operating a production machine at maximum capacity at all times with no back-up machine and no alternative for picking up the excess work is a problem waiting to happen. There are links regarding pSeries capacity planning from the Support for eServer pSeries servers page. There is information for both hardware and software capacity planning for each model of pSeries. Look in the "Plan or upgrade" tab. There is also a redbook which deals specifically with capcity planning for the pSeries. See redbooks, and search for "SG24-7071-00."

Making backups

Any time you make changes to your operating system that will affect the ODM, an operating system backup should be made. This is typically done using the mksysb command. A mksysb is handy because it can be used to boot into maintenance mode should any problem occur. It's a backup image of the important bits of the operating system, and can be used to replace a corrupted operating system image.

You should make a mksysb before doing the following things:

• adding a new hardware device
• changing tuning parameters through the "no" command.
• changing the characteristics of the operating system through smit (or by hand)
• making changes to logical volumes or volume groups
• making any change that affects the operating system or the ODM.

Always verify that you can boot from your mksysb, and that the backup is complete and uncorrupted. The mksysb command will not back up your application data. That will have to be done separately. System backups for AIX 5L versions 5.1 and 5.2 are discussed in the AIX Documentation in the "Installation Guide" or "Installation and Reference Guide." Look for the section about "System Backups." For AIX 5L version 5.3, consult the IBM eServer pSeries Information Center for AIX 5L version 5.3 Click on "AIX information." The "Installtion guides: "Installation Guide and Reference" has a whole section on creating and installing system backups.

We know it's hard to stay informed. That's why we have a system that will e-mail you when new fixes or alerts come out. You create a profile indicating what kinds of information you're interested in, so you only receive the information you're looking for. There are different types of automated information:

Subscription Services for UNIX Servers allows you to receive security advisories and alerts that maintenance fixes are available. See the website for the complete list of what you can sign up for. The basic free service is extremely helpful, but you can also choose to expand the capabilities of this service.

Offering Information is an entry point website to many, many different kinds of information within IBM, including announcements, redbooks, white papers, marketing materials, and press releases. You can search by hardware category keeping in mind that a pSeries machine is an IBM eServer UNIX server.

iSource, "Customized weekly e-news from IBM," is a nice free service which delivers information via e-mail. The e-mails have links within IBM's website for you to explore. This is a great way to keep current on new IBM websites.

Version 1.2
Last updated January 2006
Original created April 2004
For questions or comments on this handbook, please email Dawn Patterson.