The Oracle Parallel Server Option (OPS)
Pre- Aug 1999
This note summarises the introduction to a full-day tutorial on the Oracle Parallel Server Option, a.k.a. OPS. It will be addressing three questions at a fairly shallow level - what is the Oracle Parallel Server Option, why might you want to use it, and what special problems does it introduce.
The Oracle Parallel Server is a variant of the standard Oracle database software which works with a suitably enabled operating system to allow several different machines each to run their own Oracle instances which all point to the one set of Oracle database files.
It is one of the two special cases where the general rule 'one database = one instance' does not apply.
To a very large degree, the Oracle instances act independently of each other - each has its own db_block_buffer, it's own shared pool, and its own log buffer. Each even has its own set of redo log files (and archived redo log files), although every instance has to be able to see every other instance's log files and redo log files so that crash recovery can work properly.
There are only three things that the multiple instances have in common - the data files, the control files, and access to a level of the operating system which caters for clustered hardware. The operating system software that lets if all work goes under several names - cluster control software, lock manager software - whatever its name, its purpose is to allow extremely rapid co-ordination between actions on a loosely coupled set of (otherwise) independent machines
There are two reasons for choosing to run OPS - the first is high availability, the second is to cater for scalability.
Addressing the issue of high availability first - Imagine you have a OPS system with 10 nodes (i.e. 10 separate instances running) and users are connected randomly and transparently to an instance. What happens when one of your machines crashes ?
Every user connected to that machine immediately loses their connection to the database, of course, and tries to reconnect - probably they will reconnect to one of the other machines almost immediately. At the same time, one of the nine surviving instances will detect that the tenth instance has died - there will be a brief interruption of normal service as that instance takes control to sorts out any mess in the distributed lock manager and rebalance the nine survivors, and then normal service will resume. Typcially the end-users may wonder why they had a few seconds of poor response.
Compare this with a non-OPS system, running, say, the latest standby technology from Oracle. Any outstanding archived redo logs have to be applied to the standby, some on-line redo may have been lost, and the standby has to be started up. Net result - loss of time, and loss of some data. There really doesn't seem to be much competition about which is the better, and more rapid response. (But there is a downside of course - so don't leap for your cheque books).
The second issue is scalability. This is one that comes and goes, and seems to be a prime target for Parkinson's First Law (work expands to fill the time available). When you consider that it is now possible to buy an SMP box with 64 (or more) CPUs running at 400 MHz or better, and stuffed with 64 Gb of RAM, you do wonder how you could possibly manage to use up that much power and still need more to run a database. Frankly, with the increase in power of the hardware over the last year or so, there are likely to be very few reasons these days for needing OPS because of scalability issues.
However I'm sure you're familiar with the first axiom of databases: if it works, put more data into it and let more people at it. Secondly, for all the power you can put behind an Oracle database, there is still one bottleneck - the redo allocation latch. Each instance has only one redo log buffer, and the ability to generate redo can ultimately be what defines the maximum speed of your database. (There are a couple of other latches which might cave in first, but the redo is probably the most likely single point of conflict).
Sometimes you may have to run several instances at a single database because that is the only way to generate redo fast enough to do all the work that needs to be done.
Note: Oracle used to talk about OPS as the method for scaling in one of two directions - getting a large job done more quickly by sharing it between several machines, or getting more little jobs done concurrently by sharing them between several machines. The former reason for using OPS seems to have fallen by the wayside with the advent of very large SMP systems and effective use of the Parallel Query Option, and it is really only in very extreme cases of high data arrival rates (such as telecomms, and credit card tracking) that there is sufficient stress on redo generation to make the latter a necessity
In my opening comments I said that OPS basically allows several machines to run independent Oracle instances which all point to the one set of database files. Of course, these instances are not entirely independent, and that constitutes the single biggest problem when implementing an OPS solution.
An Oracle instance includes a buffer for database blocks. But if you have two instances, what do you do when they both have a copy of the same block in the buffer - what happens when both the instances want to update their own copies of the block ? Clearly the instances need some method for letting each other know what is going on and keeping each other informed of the latest state of buffered database blocks.
A more subtle version of the same problem is in handling the dictionary cache - if you have two instances you have two independent dictionary caches - so what happens when the cache on instance number 1 holds a definition of Table X and starts to parse a query against Table X - but instance number 2 executes a statement to drop Table X - again the two instances need a mechanism to keep each other informed of what is going on.
A variant of this problem then comes about in handling normal Oracle enqueues (locks). If I have two independent instances, and one takes out an exclusive table lock (type TM, mode 6), how does any other instance know that the table is locked, after all a table lock is a memory structure in the SGA, not a physical mark on the data file.
The mechanism for handling this 'cache coherency' type of problem is the DLM (or IDLM as it has become in Oracle 8), the '(Integrated) Distributed Lock Manager'. The DLM defines a locking mechanism that uses the underlying cluster management software to allow instances to 'publicise' the contents of their various caches and memory structures in a relatively efficient fashion.
The references in the manuals split DLM activity into two areas - applying it to the db block buffer Oracle refers to PCM (Parallel Cache Management) locks, and in the arena of dictionary cache management, and enqueue management etc. Oracle refers to non-PCM locks.
I will be discussing the workings of the DLM in some detail in a later article, but for the moment it is enough to say that the success or failure of an OPS system relies on understanding DLM strategies. If you understand how the DLM works, you can partition the data and the application to reduce PCM conflicts, and then implement the application using the most appropriate Oracle features to reduce the impact of non-PCM conflicts.