[pca] Cluster, zones, noreboot. - Checking System Consistency

[Don O'Malley]

Hi David,

Ronan O'Connor on our team has developed a tool called the Patch 
Pre-Flight Check tool, that will check a system is in a healthy state 
prior to beginning a maintenance session.

See http://blogs.sun.com/patch/entry/patching_pre_flight_checks_ppc for 
more details on the tool.

The tool itself can be downloaded from the Sun Patch Forum post at 
http://forums.sun.com/thread.jspa?threadID=5404908&tstart=0.
(Note: See 
http://blogs.sun.com/patch/entry/patch_forums_now_available_for for 
details of how to register for the Patch Forum.)

I've attached a copy of the ppc tool and the associated pdf manual for 
those not on registered on the Patch Forum.

Best,
-Don


David Stark wrote:
> Hi Dave.
>
> On 13/07/2010 14:30, French, David wrote:
>> I can't speak to Sun's Cluster software but here we use Veritas VCS and
>> do update on attach.  The advantage is not as much time as being able to
>> schedule the work.  For example, if you patch them all at once, they are
>> all down while this is being done.  If you migrate zones to another
>> system, leaving just the global zone on one of the systems, you can
>> patch that system.  Then when it has been rebooted and checked out, you
>> can migrate systems as needed using update on attach, until they have
>> all been updated.   Then patch the other node that has just a global and
>> roll back some of the zones to that box.
> >
>> This may cause a 2-3 reboots for some zones depending on where they
>> live, but zones usually boot fast compared to actual HW.  Just to give
>> you an idea, a system needing a hundred patches may take>  1 hr to
>> patch.   An update on attach will run in a fraction of that time.  It
>> also allows you to schedule the time with  your customer instead of
>> requiring they all be down for hours at the same time.  I find people
>> are more apt to accept quick reboots and the short time for  an update
>> on attach than accept being down for an extended period while you patch
>> everyone.
>
> Ah, excellent. Anything that reduces downtime on the zones would be a 
> win for us.
>
>> But if all zones are related then having them down at the same time may
>> not be an issue and a parallel patch may be more acceptable.
>
> Yeah, most of our clusters are single-application, but then there's 
> the dreaded 'Unix Consolidation Cluster' with 20-odd business units' 
> stuff on it. I have a feeling Update on Attach will come in handy. 
> Unfortunatley, I'll have to do an old-school patch run to get Update 
> On Attach installed :( .
>
>> Now, I will say this.  Before patching validate the current packages and
>> patches.  I mention this as I ran into an issue on one of my systems
>> (non cluster, but global and 4 container/zones) where the SUNWcsl
>> package was missing pkginfo under /var/sadm/pkg/SUNWcsl.   Not only did
>> it cause patch issues, but it also broke update on attach so much that
>> the only option I had was to rebuild the zones.  Sun (at the time)
>> wasn't much help.  The issue was deeper than just the pkginfo file but
>> also the version of that file and the files in the package used in
>> patching the zones.  The  patch utilities mangled them and I didn't
>> catch it before patching.
>
> Yeesh. Broken core libs package? Ouch.
>
>> I mention this as the version you are running is in the period I was at
>> when there were issues with the pkg/patch utilities, so better to check
>> in advance.
>
> We've been OK so far (4 clusters patched already). Fingers crossed.
>
>>     --Dave
>
> Cheers.
>
> Dave
>>
>>
>>> -----Original Message-----
>>> From: pca-bounces at lists.univie.ac.at [mailto:pca-
>>> bounces at lists.univie.ac.at] On Behalf Of David Stark
>>> Sent: Tuesday, July 13, 2010 6:09 AM
>>> To: PCA (Patch Check Advanced) Discussion
>>> Subject: Re: [pca] Cluster, zones, noreboot.
>>>
>>> Ah, sorry Filip, my question wasn't very clear.
>>>
>>> I'm wondering if the downtime for each zone would be much less using
>>> Update On Attach compared with just bringing the whole cluster down
>> and
>>> patching all the zones in parallel?
>>>
>>> On 13/07/2010 13:56, Filip Francis wrote:
>>>>
>>>> Dave,
>>>>
>>>> If you want todo parallel patchen you need to edit
>>> /etc/patch/pdo.conf
>>>> file.
>>>> That will make parallel patching to work on several zones at the
>> same
>>> time
>>>> Filip
>>>>
>>>> On 07/13/10 14:28, David Stark wrote:
>>>>> Hi all.
>>>>>
>>>>> Thanks Filip, Glen - I'd forgotten all about Update on Attach.
>>>>>
>>>>> It looks like Update On Attach is part of kernel patch 137137-09,
>>>>> which we've actually got applied on some recently patched clusters.
>>>>> The zoneadm man page lists the '-u' option on those hosts too, so
>> it
>>>>> looks like this should be usable on the next patching run.
>>>>> Since we've already got Parallel patching for zones, I'm not sure
>> if
>>>>> this will really save a lot of down time, though - is a zone
>>>>> update-attach any faster than a normal patch run?
>>>>>
>>>>> Dave
>>>>>
>>>>> On 13/07/2010 13:05, Filip Francis wrote:
>>>>>> No unless you have a certain version off Solaris i think this is
>>> only
>>>>>> from version 10u7 or 10u8 that you have this option.
>>>>>> This is his problem.
>>>>>> The cluster will not do this for you it think this is scheduled in
>>> the
>>>>>> next release of sun cluster later this year
>>>>>> Filip
>>>>>>
>>>>>>
>>>>>> On 07/13/10 13:54, Glenn Satchell wrote:
>>>>>>> When a zone migrates back to a patched system, doesn't it
>> normally
>>>>>>> update itself as it starts up the first time?
>>>>>>>
>>>>>>> Have a look at the zoneadm man page, in particular the attach and
>>>>>>> detach sub-commands. I don't know for sure, but Sun Cluster may
>> be
>>>>>>> smart enough to do the right thing and use zoneadm attach -u to
>>> bring
>>>>>>> the zone up to date when it attaches to the patched system.
>>>>>>>
>>>>>>> Perhaps a quick chat with your local Sun SE to help plan things
>>> might
>>>>>>> be time well spent?
>>>>>>>
>>>>>>> regards,
>>>>>>> -glenn
>>>>>>>
>>>>>>> On 07/13/10 21:41, David Stark wrote:
>>>>>>>> On 13/07/2010 12:31, Filip Francis wrote:
>>>>>>>>> Hi there,
>>>>>>>>
>>>>>>>> Hi!
>>>>>>>>
>>>>>>>>> I have done already quit a few upgrades of sun clusters.
>>>>>>>>> Can you give me some more details on what version off cluster +
>>>>>>>>> version
>>>>>>>>> off os.
>>>>>>>>
>>>>>>>> Sure. It's Sun Cluster 3.2 on Solaris 10 5/08. Mainly 2 node
>>> clusters,
>>>>>>>> but there's a 3 node and a 6 node as well. Mainly T5220 machines
>>>>>>>> hooked
>>>>>>>> up to EMC Symmetrix storage. The hosts' / slices are all UFS,
>> all
>>> the
>>>>>>>> zones and data filesystems are ZFS.
>>>>>>>>
>>>>>>>>> Regards
>>>>>>>>> Filip
>>>>>>>>
>>>>>>>> Cheers!
>>>>>>>> Dave
>>>>>>>>
>>>>>>>>> On 07/13/10 11:52, David Stark wrote:
>>>>>>>>>> Hi List.
>>>>>>>>>>
>>>>>>>>>> A bit off-topic, but PCA's involved, so I'm going to push my
>>> luck.
>>>>>>>>>>
>>>>>>>>>> We've got a number of Sun Cluster installs using (lots of)
>>> failover
>>>>>>>>>> zones and with no LiveUpgrade alt. boot space set up, which we
>>>>>>>>>> need to
>>>>>>>>>> patch. If we were to patch the clusters node-by-node
>> (including
>>> the
>>>>>>>>>> kernel patches that don't like zones being booted when they're
>>>>>>>>>> applied), the zones would fail over between the nodes and
>> never
>>> get
>>>>>>>>>> patched themselves, and since kernel (and some other?) patches
>>> can't
>>>>>>>>>> be applied from inside zones we would end up in a situation
>>> where
>>>>>>>>>> the
>>>>>>>>>> zones' patch databases are out of sync with the Global zone. I
>>> know
>>>>>>>>>> from very bitter experience that this is a Bad Thing, so to
>>> avoid
>>>>>>>>>> that
>>>>>>>>>> we're currently bringing the clusters down to patch them. This
>>>>>>>>>> obviously isn't optimal - people expecting 100% uptime from
>> the
>>>>>>>>>> clusters are naturally a bit annoyed at having their
>>> applications
>>>>>>>>>> down
>>>>>>>>>> for several hours while we unleash the mighty PCA.
>>>>>>>>>>
>>>>>>>>>> So, to minimise downtime I'd like to apply the noreboot
>>> patches,
>>>>>>>>>> say,
>>>>>>>>>> the night before, and have a more minimal patch run with the
>>>>>>>>>> clusters
>>>>>>>>>> down. This brings me to the question:
>>>>>>>>>>
>>>>>>>>>> Has anyone ever had any problems with noreboot patches applied
>>> to
>>>>>>>>>> live
>>>>>>>>>> systems? Any weirdness at all? I've patched plenty of test
>>>>>>>>>> machines in
>>>>>>>>>> multi-user mode, but never busy production boxes - these are
>>> fairly
>>>>>>>>>> large Oracle and SAP environments for the most part.
>>>>>>>>>>
>>>>>>>>>> Anyone with experience patching Sun Cluster care to share any
>>> top
>>>>>>>>>> tips?
>>>>>>>>>>
>>>>>>>>>> Cheers!
>>>>>>>>>>
>>>>>>>>>> Dave
>>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>>
>
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: patching_preflight_checks_RN_v1.0.1.pdf
Type: application/pdf
Size: 229478 bytes
Desc: not available
Url : https://lists.univie.ac.at/mailman/private/pca/attachments/20100713/3ae0403b/attachment-0001.pdf 
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: ppc
Url: https://lists.univie.ac.at/mailman/private/pca/attachments/20100713/3ae0403b/attachment-0001.pl