[pca] can patch 147440-09 be trusted ?

[Dennis Clarke]

> Hi Dennis,

Hi there Don.

Firstly, let me say thank you. This is the sort of response that
I admire; real thought and real communication. Oh I miss the days
of Sun. This maillist is one of the last great community old-dog
UNIX geek lists where guys like me feel comfortable. Yeah, I guess
I am a bit of a silverback in that regard. On the opensolaris
project someone once said "if you can recall building your own
kernel and it was the 80's then you're a silverback." I think
that would be me I guess.


> Let me attempt to address some of the questions you raise below.

awesome .. let's chat like it was 1994 :-)

>>I guess really the question is, can MOS and Oracle be trusted
>> to provide a kernel patch that won't make things far worse?
  [ side note : I really said that didn't I ? eek ]

> I agree that there have been issues with some of the Solaris
> 10 KUs that have been released in the past couple of months.

 I knew it wasn't just me. :-\

> Some of the more serious of these issues are a result of
> bad interactions between Solaris and third party software,
> while in other cases the issues lie squarely on Oracle's
> shoulders.

> The biggest single issue that we have hit in the past
> couple of months is that the Solaris 10 Update 10 "Feature"
> Kernel patch (145900-19) changed private Solaris interfaces.

OKay, well that's cool in my opinion. Private means private and
people should never rely on them. That is badly written software
on someone elses fault, not Oracle and sure as hell no one from
Sun.

> These changes led to third party software that used these
> private Solaris interfaces to break.

Big mistake on their part right?

You know, if you put a red button on a wall in a waiting room
with a sign under it that says "danger - do not press" there
will always be some turkey ( and his manager ) that walk up and
press the thing. Just to see what happens. Then stand there
looking all glad and happy when nothing bad *seems* to happen.
Meanwhile six floors down in the basement in a service room
with crazy looking old pipes, wires and somewhat crusty equipment
the accounting departments PDP-11 ( you know, they could never
migrate because Jenkins wrote it on COBOL in 1974 and he drank
scotch at work but wrote great code. He retired in 1992. ) begins
a backup cycle in the middle of the day. Then the cheque printer
up on the twelfth floor stops printing and that old impact printer
thing has been on maintenance for thirty years with IBM but the
new IT manager dropped the contract to cut corners. Well that
is what happens when you depend on private interfaces. Bad stuff
and you may not even see it. Not yet. But just wait a sec and
it will hit you. Yep. Some turkey and his manager just loves to
press that button.

Soory, silverbacks sometime rant. Ummm .. you were sayin' ?

> The most common issue that customers reported as a result of
> this issue was that systems running EMC PowerPath and Solaris
> 10 experienced a system panic if 144500-19 was installed.
> This is detailed in SunAlert 1358671.1

Gotta love it!  Way to go EMC. You know, this is why I like the
6540 StorageTek array controllers and Brocade switches. Never a
problem and those things just work and rock. HP EVA? Well, yeah,
if you hide it behind a fibre switch and keep a guy on staff to
talk to it and the HP contract boys.

> EMC have since released a patch for PowerPath to fix this
> specific problem.

  ... and go call Jenkins who retired in 1992 to rewrite some
routines. Oh, and fire that manager and promote the above
mentioned turkey who argued with him.

> Similar issues have been reported with other third party
> multipathing solutions like Hitachi HDS, Falconstor and Dynapath.

 * snort *

Lovely.

> I am not saying that Oracle are blameless in the Kernel patching
> issues. For example, there was a change made in patch 147440-04,
> that broke systems using a Veritas (VxVM) swap or dump device
> volume. As a result patch 147440-04 was withdrawn from MOS and
> SunAlert 1374089.1 was issued.

Sounds like it has been fun dancing in a minefield in there.
So then, you keep scotch at your desk now?  [ I sure do ]


> There have been other issues with KU 147440-xx that only affect
> systems running very old firmware versions, or other edge cases.
> Examples of this are SunAlerts 1365975.1, 1359916.1 & 1380247.1.

wow

> I agree that there have certainly been more some issues with
> the KU patch in the past couple of months than usual, the release
> of 147440-08 fixed the last major outstanding issue (the VxVM
> issue detailed in 1374089.1).

Have these people not heard of ZFS ? Yet ?

> In any case where Oracle know (or are made aware) of a patch
> issue the following happens:
[side note : formatting of good ol' ASCII by me ]

    - We add a note to the "Special Install Instructions"
      section of the patch to warn customers of the issue ASAP.

    - If appropriate we will also release a SunAlert to detail
     "Data Loss" or "System Availability" issues.

    - If the issue is severe we will withdraw a patch.

That last one can really really be scary. That is when the hair on
the back of my neck goes up as you just know something really bad happened.


> If you are looking to assess the likelihood of running into an
> issue as a result of installing a patch (particularly the KU)
> there are a number of things you can/should do:

K, I'll reach for the scotch early today and read. What follows
may be motherhood but it is good stuff.


> Read the latest version of README, specifically the "Special
> Install Instructions" - eg.
> https://updates.oracle.com/Orion/Services/download?type=readme&bugfix_name=147440-08
> Look up the patch on the flash version of MOS (if possible)
[side note : I never use the flash version. Ever. Gee, ain't HTML
 4.01 and CSS good enough ? ]

> Personally I'm not a fan of the flash-based MOS, but there are a
> couple of really useful features:

I'll set aside some of the 18 year old Macallan for you.
I hate flash sites.

> The "Related Knowledge to this patch" will list any SunAlerts
> that contain that patchid. (This means that issues that are
> either caused by or fixed in this patch will be detailed here.)

Well, can I say "maybe detailed". Some stuff is locked away and
hidden and carries a description like "problem with NFS"".

> There is a "Community Discussion" on the right of the patch
> description. This gives customers the opportunity to provide
> feedback on any issues they have experienced with that specific
> patch. This feedback flows into the moderated Oracle "Patch
> Reviews - SUN" Community and is also displayed on a per patch
> basis in this "Community Discussion" frame.

Comments like, "Dear Oracle, the entire 17th floor just stopped
 working. I'm writing a resume. Thanks." That sort of thing?   ;-)

>> The reason I ask, and would love to hear from others that
>> patch multiple tiers of servers ( you know, production,
>> testing, development etc etc ), is that I actually read the
>> kernel patch README's. In the past that seemed to be a good
>> policy. I could actually see the bugids affected and then
>> determine if it was reasonable to apply the patch on the
>> test level servers or rush to get it into the prod level.
>> Patch 147440-09 is less than thrilling.

>> Here is the bugid list in the README :

>>   6878961 problem with NFS
>>   6927023 need support for AMD family 15h processor
>>   6932919 need support for AMD's family 15h ISA enhancements
>>   7030516 hpet_acpi_init() should be called only if hpet
>>           is really required
>>   7047435 'genunix: WARNING: preconfig failed: disk' when
>>            configure hard disk drive for removal
>>   7105132 deadman panic on a T3-1

>> Well gee, "problem with NFS" means what exactly? Does it mean
>> that unless you apply this patch you have a security issue or
>> a pending kernel panic and core dump? The other bugids apply
>> support for the bulldozer AMD chip and that's a good thing
>> since HP has been shipping those for a while.

>> Would be cool to see what this "problem with NFS" is all about.
>> However when I login to MOS that bugid is not even listed. See
>> image attached. I can see this bugid :
>>
>>https://supporthtml.oracle.com/ep/faces/secure/km/BugDisplay.jspx?id=7047435
>>
>>Which is in the README but NOT on the MOS page for this kernel
>> patch.

>> If I try to see
>>https://supporthtml.oracle.com/ep/faces/secure/km/BugDisplay.jspx?id=6878961
>>I get a warm and fuzzy "The Bug ID is invalid or you do not
>> have permission to view the bug". The same goes for bugid
>> 6382683 which is listed on the MOS page but NOT in the README.
>>
>>Seriously, can Oracle be trusted or is this a case of "trust
>> us, we know, you don't, and you don't need to know."
>>Because that really does not fly well with me.

> Ok, so the above discussion is all related to the patch
> content (i.e. what's being fixed).

 * sigh *

> While I understand your frustration here (and admire your
> attention to detail), what you are experiencing is not new
> under Oracle. This issue is related to security fixes (though
> there may be rare cases where other non-security bugs that
> are not publicly visible either).

Sorry, but "problem with NFS" doesn't help me at all.

> Before the Oracle acquisition, Sun never published bug
> reports for any security issues.

I must be thinking OpenSolaris.  ;-)

>(Back in the old SunSolve days we would release a "HTML"
>version of a patch README very similar to
>https://updates.oracle.com/Orion/Services/download?type=readme&>amp;bugfix_name=147440-08
>that would contain links to bug reports for non-security bugs
> only.)

I miss the old old Sunsolve days. Back when a single phone call got you really
awesome support on Solaris 2.5.1 or similar.

> In addition, the Bug description in the README for a patch was
> (and still is) reviewed by the security team to ensure that it
> is not providing enough information on any security-related
> issues to hackers looking to reverse-Engineer security
> vulnerabilities into potential attacks. The idea is that you
> get enough information to decide if you use the particular
> technology

  like NFS ? Gee. That is like saying "do you inhale air often?"

> and if you feel it is appropriate for you to install
> the patch. It's far from perfect, but with security issues
> we must err on the side of caution.

[ I just reached for the scotch again ]

> For this reason we do not (and never had) made bug reports for
> security issues publicly available.

Thanks for being here Don. You know we love you. This is what we
have at the moment and I only worry, how far into the minefield
are you?  The last six yards ahead or are you in the middle and
still dancing?

I know, tough question and you can't do much. Hell, we love that
you are in this maillist as PCA has become de Facto, de Rigueur,
de bomb.

Dennis
ps: seee image attached

-- 
--
http://pgp.mit.edu:11371/pks/lookup?op=vindex&search=0x1D936C72FA35B44B
+-------------------------+-----------------------------------+
| Dennis Clarke           | Solaris and Linux and Open Source |
| dclarke at blastwave.org   | Respect for open standards.       |
+-------------------------+-----------------------------------+
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Desk_Supplies_KU-scotch.png
Type: image/png
Size: 130422 bytes
Desc: not available
URL: <https://lists.univie.ac.at/pipermail/pca/attachments/20120110/9dba1540/attachment-0001.png>