[pca] interesting patch problem with Oracle Access Manager Version: 11.1.1.5.0

Tue Jul 16 15:50:23 CEST 2013

In the past, I've been bitten by every possible thing that could go wrong at one time or another.
I've developed contingencies that avoid these problems as best as possible -- which I've learned over the years.

One of my past issues was not being able to get patches onto a prod box in a timely manner. Usually due to an external issue out of my control.
The process I follow now makes this problem impossible -- because the patches are staged in advance.

Sorry about the length of this post. Hopefully something I say here helps someone.

Here is the general flow I follow for patching...
0) Download the patches from Oracle to a shared NFS mount point. (./pca --download ...)
1) Install patches onto a VM/host that no-one uses, just to see if it breaks really badly. (such as not booting up)
2) back-out the patches to see if the back-out fails. (using live update to activate previous boot environment)
3) re-install the same patches again to same VM/host.
4) Install patches from shared NFS mount point onto all Lab/DEV boxes.
5) Stage patches from step #0 to QA/DR via SFTP.
6) Install patches in QA/DR
7) Stage patches from step #0 to Prod via SFTP.
8) Install patches in Prod.

As soon as step #4 has been tested, we know that the patches are good, and the list of patches is "frozen" in a folder as a bunch of .zip files.
We can then do steps #5 and #7 right away -- which may be several weeks before #6 and #8 occur.

Points to note: Prod servers only allow incoming port 22, not outbound 80 and 443, so therefore they can't use a http caching proxy server to get patches.
The patch's .zip file only goes across the WAN once via SFTP per datacenter. Then, from that one point is copied via SFTP to each host over LAN.
Also, only step #0 uses the Internet to access Oracle's web site. Step #0 uses "pca --fromfiles=DIR ..." multiple times, once for each host. This way we get all needed patches for all hosts at one time in advance. Later in steps #6 and #8 PCA does not need to pull anything via port 80/443 because all patches are already in the --patchdir=DIR location.

If new patches become available after step #1 and before step #8, they are NOT included in the patches installed in step #8.
For steps #1-8 above, we use the "pca --nocheckxref --xrefdir=DIR --patchdir=DIR" where "DIR" is the same location as the patches stored in step #0, 5, and 7.
This prevents PCA from changing the list of patches after they are "frozen".

Neil G. Brookins
Identity and Authentication Solutions - IT Global Solutions
Towers Watson
1500 Market Street | Philadelphia, PA 19102
Phone: +1 215 246 6046
neil.brookins at towerswatson.com

-----Original Message-----
From: pca-bounces at lists.univie.ac.at [mailto:pca-bounces at lists.univie.ac.at] On Behalf Of Paul Lanken
Sent: Monday, July 15, 2013 11:43 AM
To: PCA (Patch Check Advanced) Discussion
Subject: Re: [pca] interesting patch problem with Oracle Access Manager Version: 11.1.1.5.0

On Mon, Jul 15, 2013 at 8:33 AM, Martin Paul <martin.paul at univie.ac.at> wrote:
> Am 15.07.2013 14:23, schrieb Jan Holzhueter:
>
>> So I had to cleanup my proxy to not have bad patches. (Maybe some check
>> if the downloaded file is a zipfile would be nice.)
>
>
> There is a check in the pca code, but it seems as if it failed in this case.
>
>
>> Looks like it was a temp problem. This morning after the cleanup the
>> download of the broken patches went fine. (Not sure about the number.
>> But was more then 10.)
>
>
> Thanks for the report, good to hear that!

I can confirm that the problem seemed to simply "go away" all on its
own sometime
last night.  I did have a downtime window for Saturday night but
missed it entirely
because I could not get the patches in a reasonable fashion.  Something odd and
clearly, based on this maillist, an obscure issue never before seen.

So please let me share an idea, because I found myself in a sysadmin situation
where I had a downtime window approved on production hardware and was watching
it slip past.  Felt a bit of stress that night however Neil G.
Brookins, ( see this thread )
changed my life with his posting.  I was wondering about what to do with some of
the bigger Niagara servers which have thirty zones or more.  Normally
I get a downtime
window starting Friday night and run all weekend to apply patches. Stupid me.

Regardless, I was stuck a bit. Saw a few patches being downloaded and
then a whole
stack of files that all looked like NNNNNN-RR.zip ( where N and R are
digits ) but the
files were just html error messages.  I thought I could sed/awk/cut/tr
script-fu my way
around this and send the PCA process into a loop until I had all the
patches. I even
wrote a script to get the list of missing patches first and then post
process that with
awk to create another script which used PCA to pull down the patches
one at a time
with a "sleep 30" in between each call to PCA.  Ugly but it also did not help.

So there I was, coffee cup in hand.  Loiking at the clock and then an
idea struck me.

- idea to use patchdiag.CHECKSUMS file to verify patch integrity

I am sure you have heard it all before.  Hardly new.  There is a
public file available
at this url :  ( from my panicy scriptint that night )

    # do not pass user and passwd for this ... it is public
    # Be silent about this. The info is nice to have and not a need at
this time.
    $PCA_WGET --output-file=/dev/null --quiet --continue
--no-check-certificate \
      --output-document=$PCA_XREFDIR/patchdiag.CHECKSUMS \
      https://getupdates.oracle.com/reports/CHECKSUMS > /dev/null 2>&1

    if [ ! -f $PCA_XREFDIR/patchdiag.CHECKSUMS ]; then
        /bin/printf "WARN : patchdiag.CHECKSUMS could not be fetched.\n"
    fi

So then in that file, for any given patch ( I guess ) there is an MD5
hash but it
is on a separate line from the patch filename.

So for this patch :

    148931-03 Jul/12/13 Oracle Solaris Studio 12.3: Patch for
Performance Library

I see this :

# /usr/xpg4/bin/grep -n "148931-03" pca_data/xref/patchdiag.CHECKSUMS
366421:148931-03.zip

# cat -n pca_data/xref/patchdiag.CHECKSUMS | head -366424 | tail -4
366421  148931-03.zip
366422  MD5: b848434ebf59e507e347cb99ff7eb1ba
366423  SysV Sum: 63367   256161
366424  Sum: 46010   256161

So some script-fu with sed and tr etc etc can get me the MD5 hash.

.... and down the rabbit hole I go to verify the patches etc etc.

I am sure you have heard it all before.

Paul

Notice of Confidentiality
This transmission contains information that may be confidential. It has been prepared for the sole and exclusive use of the intended recipient and on the basis agreed with that person. If you are not the intended recipient of the message (or authorized to receive it for the intended recipient), you should notify us immediately; you should delete it from your system and may not disclose its contents to anyone else.

This e-mail has come to you from Towers Watson Delaware Inc. or Towers Watson Pennsylvania Inc.