[pca] Cluster, zones, noreboot.

[David Stark]

Hi Dave.

On 13/07/2010 14:30, French, David wrote:
> I can't speak to Sun's Cluster software but here we use Veritas VCS and
> do update on attach.  The advantage is not as much time as being able to
> schedule the work.  For example, if you patch them all at once, they are
> all down while this is being done.  If you migrate zones to another
> system, leaving just the global zone on one of the systems, you can
> patch that system.  Then when it has been rebooted and checked out, you
> can migrate systems as needed using update on attach, until they have
> all been updated.   Then patch the other node that has just a global and
> roll back some of the zones to that box.
 >
> This may cause a 2-3 reboots for some zones depending on where they
> live, but zones usually boot fast compared to actual HW.  Just to give
> you an idea, a system needing a hundred patches may take>  1 hr to
> patch.   An update on attach will run in a fraction of that time.  It
> also allows you to schedule the time with  your customer instead of
> requiring they all be down for hours at the same time.  I find people
> are more apt to accept quick reboots and the short time for  an update
> on attach than accept being down for an extended period while you patch
> everyone.

Ah, excellent. Anything that reduces downtime on the zones would be a 
win for us.

> But if all zones are related then having them down at the same time may
> not be an issue and a parallel patch may be more acceptable.

Yeah, most of our clusters are single-application, but then there's the 
dreaded 'Unix Consolidation Cluster' with 20-odd business units' stuff 
on it. I have a feeling Update on Attach will come in handy. 
Unfortunatley, I'll have to do an old-school patch run to get Update On 
Attach installed :( .

> Now, I will say this.  Before patching validate the current packages and
> patches.  I mention this as I ran into an issue on one of my systems
> (non cluster, but global and 4 container/zones) where the SUNWcsl
> package was missing pkginfo under /var/sadm/pkg/SUNWcsl.   Not only did
> it cause patch issues, but it also broke update on attach so much that
> the only option I had was to rebuild the zones.  Sun (at the time)
> wasn't much help.  The issue was deeper than just the pkginfo file but
> also the version of that file and the files in the package used in
> patching the zones.  The  patch utilities mangled them and I didn't
> catch it before patching.

Yeesh. Broken core libs package? Ouch.

> I mention this as the version you are running is in the period I was at
> when there were issues with the pkg/patch utilities, so better to check
> in advance.

We've been OK so far (4 clusters patched already). Fingers crossed.

> 	--Dave

Cheers.

Dave
>
>
>> -----Original Message-----
>> From: pca-bounces at lists.univie.ac.at [mailto:pca-
>> bounces at lists.univie.ac.at] On Behalf Of David Stark
>> Sent: Tuesday, July 13, 2010 6:09 AM
>> To: PCA (Patch Check Advanced) Discussion
>> Subject: Re: [pca] Cluster, zones, noreboot.
>>
>> Ah, sorry Filip, my question wasn't very clear.
>>
>> I'm wondering if the downtime for each zone would be much less using
>> Update On Attach compared with just bringing the whole cluster down
> and
>> patching all the zones in parallel?
>>
>> On 13/07/2010 13:56, Filip Francis wrote:
>>>
>>> Dave,
>>>
>>> If you want todo parallel patchen you need to edit
>> /etc/patch/pdo.conf
>>> file.
>>> That will make parallel patching to work on several zones at the
> same
>> time
>>> Filip
>>>
>>> On 07/13/10 14:28, David Stark wrote:
>>>> Hi all.
>>>>
>>>> Thanks Filip, Glen - I'd forgotten all about Update on Attach.
>>>>
>>>> It looks like Update On Attach is part of kernel patch 137137-09,
>>>> which we've actually got applied on some recently patched clusters.
>>>> The zoneadm man page lists the '-u' option on those hosts too, so
> it
>>>> looks like this should be usable on the next patching run.
>>>> Since we've already got Parallel patching for zones, I'm not sure
> if
>>>> this will really save a lot of down time, though - is a zone
>>>> update-attach any faster than a normal patch run?
>>>>
>>>> Dave
>>>>
>>>> On 13/07/2010 13:05, Filip Francis wrote:
>>>>> No unless you have a certain version off Solaris i think this is
>> only
>>>>> from version 10u7 or 10u8 that you have this option.
>>>>> This is his problem.
>>>>> The cluster will not do this for you it think this is scheduled in
>> the
>>>>> next release of sun cluster later this year
>>>>> Filip
>>>>>
>>>>>
>>>>> On 07/13/10 13:54, Glenn Satchell wrote:
>>>>>> When a zone migrates back to a patched system, doesn't it
> normally
>>>>>> update itself as it starts up the first time?
>>>>>>
>>>>>> Have a look at the zoneadm man page, in particular the attach and
>>>>>> detach sub-commands. I don't know for sure, but Sun Cluster may
> be
>>>>>> smart enough to do the right thing and use zoneadm attach -u to
>> bring
>>>>>> the zone up to date when it attaches to the patched system.
>>>>>>
>>>>>> Perhaps a quick chat with your local Sun SE to help plan things
>> might
>>>>>> be time well spent?
>>>>>>
>>>>>> regards,
>>>>>> -glenn
>>>>>>
>>>>>> On 07/13/10 21:41, David Stark wrote:
>>>>>>> On 13/07/2010 12:31, Filip Francis wrote:
>>>>>>>> Hi there,
>>>>>>>
>>>>>>> Hi!
>>>>>>>
>>>>>>>> I have done already quit a few upgrades of sun clusters.
>>>>>>>> Can you give me some more details on what version off cluster +
>>>>>>>> version
>>>>>>>> off os.
>>>>>>>
>>>>>>> Sure. It's Sun Cluster 3.2 on Solaris 10 5/08. Mainly 2 node
>> clusters,
>>>>>>> but there's a 3 node and a 6 node as well. Mainly T5220 machines
>>>>>>> hooked
>>>>>>> up to EMC Symmetrix storage. The hosts' / slices are all UFS,
> all
>> the
>>>>>>> zones and data filesystems are ZFS.
>>>>>>>
>>>>>>>> Regards
>>>>>>>> Filip
>>>>>>>
>>>>>>> Cheers!
>>>>>>> Dave
>>>>>>>
>>>>>>>> On 07/13/10 11:52, David Stark wrote:
>>>>>>>>> Hi List.
>>>>>>>>>
>>>>>>>>> A bit off-topic, but PCA's involved, so I'm going to push my
>> luck.
>>>>>>>>>
>>>>>>>>> We've got a number of Sun Cluster installs using (lots of)
>> failover
>>>>>>>>> zones and with no LiveUpgrade alt. boot space set up, which we
>>>>>>>>> need to
>>>>>>>>> patch. If we were to patch the clusters node-by-node
> (including
>> the
>>>>>>>>> kernel patches that don't like zones being booted when they're
>>>>>>>>> applied), the zones would fail over between the nodes and
> never
>> get
>>>>>>>>> patched themselves, and since kernel (and some other?) patches
>> can't
>>>>>>>>> be applied from inside zones we would end up in a situation
>> where
>>>>>>>>> the
>>>>>>>>> zones' patch databases are out of sync with the Global zone. I
>> know
>>>>>>>>> from very bitter experience that this is a Bad Thing, so to
>> avoid
>>>>>>>>> that
>>>>>>>>> we're currently bringing the clusters down to patch them. This
>>>>>>>>> obviously isn't optimal - people expecting 100% uptime from
> the
>>>>>>>>> clusters are naturally a bit annoyed at having their
>> applications
>>>>>>>>> down
>>>>>>>>> for several hours while we unleash the mighty PCA.
>>>>>>>>>
>>>>>>>>> So, to minimise downtime I'd like to apply the noreboot
>> patches,
>>>>>>>>> say,
>>>>>>>>> the night before, and have a more minimal patch run with the
>>>>>>>>> clusters
>>>>>>>>> down. This brings me to the question:
>>>>>>>>>
>>>>>>>>> Has anyone ever had any problems with noreboot patches applied
>> to
>>>>>>>>> live
>>>>>>>>> systems? Any weirdness at all? I've patched plenty of test
>>>>>>>>> machines in
>>>>>>>>> multi-user mode, but never busy production boxes - these are
>> fairly
>>>>>>>>> large Oracle and SAP environments for the most part.
>>>>>>>>>
>>>>>>>>> Anyone with experience patching Sun Cluster care to share any
>> top
>>>>>>>>> tips?
>>>>>>>>>
>>>>>>>>> Cheers!
>>>>>>>>>
>>>>>>>>> Dave
>>>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>
>
>