Skip Headers
Oracle® TimesTen In-Memory Database TimesTen to TimesTen Replication Guide
Release 11.2.1

Part Number E13072-04
Go to Documentation Home
Home
Go to Book List
Book List
Go to Table of Contents
Contents
Go to Index
Index
Go to Master Index
Master Index
Go to Feedback page
Contact Us

Go to previous page
Previous
Go to next page
Next
View PDF

6 Using Oracle Clusterware to Manage Active Standby Pairs

Oracle Clusterware monitors and controls applications to provide high availability. This chapter describes how to use Oracle Clusterware to manage availability for a TimesTen active standby pair.

For more information about Oracle Clusterware, see Oracle Clusterware Administration and Deployment Guide in the Oracle Database documentation.

This chapter includes the following topics:

Overview

Figure 6-1 shows an active standby pair with one read-only subscriber in the same local network. The active data store, the standby data store and the read-only subscriber are on different nodes. There are two nodes that are not part of the active standby pair that are also running TimesTen. An application updates the active data store. An application reads from the standby and the subscriber. All of the nodes are connected to shared storage.

Figure 6-1 Active standby pair with one subscriber

Description of Figure 6-1 follows
Description of "Figure 6-1 Active standby pair with one subscriber"

You can use Oracle Clusterware to start, monitor and automatically fail over TimesTen applications in response to node failures and other events. See "Planned maintenance" and "Recovering from failures" for details.

Oracle Clusterware can be implemented at two levels of availability for TimesTen. The basic level of availability manages two master nodes and up to 127 read-only subscriber nodes in the cluster. The active standby pair is defined with physical host names and IP addresses. If both master nodes fail, user intervention is necessary to migrate the active standby scheme to new hosts. When both master nodes fail, Oracle Clusterware notifies the user.

The advanced level of availability uses virtual IP addresses for the active, standby and read-only subscriber data stores. Extra nodes can be included in the cluster that are not part of the initial active standby pair. If a failure occurs, the use of virtual IP addresses allows one of the extra nodes to take on the role of a failed node automatically.

If your applications connect to TimesTen in a client/server configuration, automatic client failover enables the client to reconnect automatically to the master data store with the active role after a failure. See "Working with the TimesTen Client and Server" in Oracle TimesTen In-Memory Database Operations Guide and "TTC_FailoverPortRange" in Oracle TimesTen In-Memory Database Reference.

The ttCWAdmin utility is used to administer TimesTen active standby pairs in a cluster that is managed by Oracle Clusterware. The configuration for each active standby pair is stored in an initialization file called cluster.oracle.ini by default. The information in this file is used to create Oracle Clusterware resources. Resources are used to manage each TimesTen daemon, data store, TimesTen processes, user applications and virtual IP addresses. For more information about the ttCWAdmin utility, see "ttCWAdmin" in Oracle TimesTen In-Memory Database Reference. For more information about the cluster.oracle.ini file, see "The cluster.oracle.ini file".

Active standby configurations

Use Oracle Clusterware to manage only these configurations:

  • Active standby pair with or without read-only subscribers

  • Active standby pair (with or without read-only subscribers) with AWT cache groups

  • Active standby pair (with or without read-only subscribers) with read-only cache groups

  • Active standby pair (with or without read-only subscribers) with AWT cache groups and read-only cache groups

Required privileges

The user must be the TimesTen instance administrator and must belong to the same operating system group as the Oracle Clusterware installation user.

On UNIX and Linux platforms, these ttCWAdmin commands must be executed by the root user:

ttCWAdmin -createVIPs

ttCWAdmin -dropVIPs

Hardware and software requirements

See Oracle Clusterware Administration and Deployment Guide for network and storage requirements and information about Oracle Clusterware configuration files.

Oracle Clusterware and TimesTen should be installed in the same location on all nodes.

All machines should use Network Time Protocol (NTP) or a similar system to remain within 250 milliseconds of each other. This prevents Oracle Clusterware from unnecessarily restarting machines.

Restricted commands and SQL statements

When you use Oracle Clusterware to manage an active standby pair, do not use these commands and SQL statements:

  • CREATE ACTIVE STANDBY PAIR, ALTER ACTIVE STANDBY PAIR and DROP ACTIVE STANDBY PAIR SQL statements

  • ttAdmin and ttRepAdmin -duplicate utilities

  • ttRepStart and ttRepStop built-in procedures

In addition, do not call ttDaemonAdmin -stop before calling ttCWAdmin -shutdown.

The TimesTen integration with Oracle Clusterware accomplishes these operations with the ttCWAdmin utility and the attributes in the cluster.oracle.ini file. If you use these commands and statements, Oracle Clusterware will not operate properly.

For more information about the built-ins and utilities, see Oracle TimesTen In-Memory Database Reference. For more information about the SQL statements, see Oracle TimesTen In-Memory Database SQL Reference.

The cluster.oracle.ini file

The configuration for each active standby pair is stored in an initialization file called cluster.oracle.ini. The information in this file is used to create Oracle Clusterware resources that manage TimesTen daemons, data stores, TimesTen processes, user applications and virtual IP addresses.

The user creates the cluster.oracle.ini file as a text file and places it in the daemon home directory on the host for the active data store. By default this directory is:

All of the attributes that can be used in the cluster.oracle.ini file are described in Chapter 7, "TimesTen Attributes for Oracle Clusterware".

The entry name in the cluster.oracle.ini file must be the same as an existing DSN:

For example, [basicDSN] in the cluster.oracle.ini file described in "Configuring basic availability" must exist in the sys.odbc.ini file.

This section includes sample cluster.oracle.ini files for these configurations:

Configuring basic availability

This example shows an active standby pair with no subscribers. The hosts for the active data store and the standby data store are host1 and host2. The list of hosts is delimited by commas. You can include spaces for readability if desired.

The ttCWAdmin utility is used to administer TimesTen active standby pairs in a cluster that is managed by Oracle Clusterware. When the ttCWAdmin utility creates a profile for the active standby pair, it creates scripts that are placed in the location specified by the ScriptInstallDir attribute.

[basicDSN]
MasterHosts=host1,host2
ScriptInstallDir=/mycluster/TTscripts

This is an example of a cluster.oracle.ini file for an active standby pair with one subscriber on host3:

[basicSubscriberDSN]
MasterHosts=host1,host2
SubscriberHosts=host3
ScriptInstallDir=/mycluster/TTscripts

On Windows, the format of the ScriptInstallDir attribute is similar to this:

ScriptInstallDir=c:\mycluster\TTscripts

Configuring advanced availability

In this example, the hosts for the active data store and the standby data store are host1 and host2. host3 and host4 are extra nodes that can be used for failover. There are no subscriber nodes. MasterVIP specifies the virtual IP addresses defined for the master data stores. VIPInterface is the name of the public network adaptor. VIPNetMask defines the netmask of the virtual IP addresses.

[advancedDSN]
MasterHosts=host1,host2,host3,host4
ScriptInstallDir=/mycluster/TTscripts
MasterVIP=192.168.1.1, 192.168.1.2
VIPInterface=eth0
VIPNetMask=255.255.255.0

This example has one subscriber on host4. There is one extra node that can be used for failing over the master data stores and one extra node that can be used for the subscriber data store. MasterVIP and SubscriberVIP specify the virtual IP addresses defined for the master and subscriber data stores. VIPInterface is the name of the public network adaptor. VIPNetMask defines the netmask of the virtual IP addresses.

[advancedSubscriberDSN]
MasterHosts=host1,host2,host3
SubscriberHosts=host4,host5
ScriptInstallDir=/mycluster/TTscripts
MasterVIP=192.168.1.1, 192.168.1.2
SubscriberVIP=192.168.1.3
VIPInterface=eth0
VIPNetMask=255.255.255.0

Ensure that the extra nodes:

  • Have TimesTen installed

  • Have the direct-linked application installed if this is part of the configuration. See "Implementing application failover".

  • Have a copy of the cluster.oracle.ini file in the same location as the other hosts.

Including cache groups in the active standby pair

If the active standby pair replicates one or more AWT or read-only cache groups, set the CacheConnect attribute to y.

This example specifies an active standby pair with one subscriber in an advanced availability configuration. The active standby pair replicates one or more cache groups.

[advancedCacheDSN]
MasterHosts=host1,host2,host3
SubscriberHosts=host4, host5
ScriptInstallDir=/mycluster/TTscripts
MasterVIP=192.168.1.1, 192.168.1.2
SubscriberVIP=192.168.1.3
VIPInterface=eth0
VIPNetMask=255.255.255.0
CacheConnect=y

Implementing application failover

TimesTen integration with Oracle Clusterware can facilitate the failover of an application that is linked to any of the data stores in the active standby pair. Both direct-linked and client/server applications that are on the same machine as Oracle Clusterware and TimesTen can be managed.

The required attributes in the cluster.oracle.ini file for failing over an application are:

  • AppName - Name of the application to be managed by Oracle Clusterware

  • AppStartCmd - Command line for starting the application

  • AppStopCmd - Command line for stopping the application

  • AppCheckCmd - Command line for executing an application that checks the status of the application specified by AppName

  • AppType - Determines the data store to which the application is linked. The possible values are Active, Standby, and Subscriber[index].

Optionally, you can also set AppFailureThreshold, AppFailoverDelay, and AppScriptTimeout. These attributes have default values.

Oracle Clusterware uses the user-supplied script or program specified by AppCheckCmd to monitor the application. The script that checks the status of the application must be written to return 0 for success and a nonzero number for failure. When Oracle Clusterware detects a nonzero value, it takes action to recover the failed application.

This example shows advanced availability configured for an active standby pair with with no subscribers. The reader application is an application that queries the data in the standby data store.

[appDSN]
MasterHosts=host1,host2,host3,host4
ScriptInstallDir=/mycluster/TTscripts
MasterVIP=192.168.1.1, 192.168.1.2
VIPInterface=eth0
VIPNetMask=255.255.255.0
AppName=reader
AppType=Standby
AppStartCmd=/mycluster/reader/app_start.sh
AppStopCmd=/clusterware/reader/app_stop.sh
AppCheckCmd=/clusterware/reader/app_check.sh

AppStartCmd, AppStopCmd and AppCheckCmd can include arguments. For example, consider a valid cluster.oracle.ini file on Windows. In this example the application is directly linked to the active data store. The script for starting, stopping, and checking the application takes arguments for the DSN and the action to take (-start, -stop and -check).

Note the double quotes for the specified paths in AppStartCmd, AppStopCmd and AppCheckCmd. The quotes are needed because there are spaces in the path.

[appWinDSN]
MasterHosts=host1,host2,host3,host4
ScriptInstallDir=C:\Cluster Files\TTscripts
MasterVIP=192.168.1.1, 192.168.1.2
VIPInterface=Local Area Connection
VIPNetMask=255.255.255.0
AppName=UpdateApp
AppType=Active
AppStartCmd="C:\Program Files\UserApps\UpdateApp.exe" -dsn myDSN -start
AppStopCmd= "C:\Program Files\UserApps\UpdateApp.exe" -dsn myDSN -stop
AppCheckCmd="C:\Program Files\UserApps\UpdateApp.exe" -dsn myDSN -check

You can configure failover for more than one application. Use AppName to name the application and provide values for AppType, AppStartCmd, AppStopCmd and AppCheckCmd immediately following the AppName attribute. You can include blank lines for readability. For example:

[app2DSN]
MasterHosts=host1,host2,host3,host4
ScriptInstallDir=/mycluster/TTscripts
MasterVIP=192.168.1.1, 192.168.1.2
VIPInterface=eth0
VIPNetMask=255.255.255.0

AppName=reader
AppType=Standby
AppStartCmd=/mycluster/reader/app_start.sh
AppStopCmd=/clusterware/reader/app_stop.sh
AppCheckCmd=/clusterware/reader/app_check.sh

AppName=update
AppType=Active
AppStartCmd=/mycluster/update/app2_start.sh
AppStopCmd=/clusterware/update/app2_stop.sh
AppCheckCmd=/clusterware/update/app2_check.sh

Recovering from failure of both master nodes

If both master nodes fail, Oracle Clusterware can automatically recover the master data stores to two new nodes. Automatic recovery requires:

  • Advanced availability (virtual IP addresses and at least four hosts)

  • No cache groups on the active standby pair data stores

  • AutoRecover set to y

  • RepBackupDir specifies a directory on shared storage

  • RepBackupPeriod must be set to a value greater than 0

TimesTen first performs a full backup of the active data store and then performs incremental backups. You can specify the optional attribute RepfullbackupCycle to manage when TimesTen performs subsequent full backup. By default, TimesTen performs a full backup after every five incremental backups.

If RepBackupDir and RepBackupPeriod are configured for backups, TimesTen performs backups for any master data store that becomes active. It does not delete backups that were performed for a data store that used to be active and has become the standby unless the data store becomes active again. Ensure that the shared storage has enough space for two complete data store backups. ttCWAdmin -restore automatically chooses the correct backup files.

TimesTen uses the ttBackup utility to perform backups. Incremental backups increase the amount of log records in the transaction log files. Ensure that the values of RepBackupPeriod and RepfullbackupCycle are small enough to prevent a large amount of log records in the transaction log file. For more information about the interaction between backups and log records, see "ttBackup" in Oracle TimesTen In-Memory Database Reference.

This example shows attribute settings for automatic recovery.

[autorecoveryDSN]
MasterHosts=host1,host2,host3,host4
ScriptInstallDir=/mycluster/TTscripts
MasterVIP=192.168.1.1, 192.168.1.2
VIPInterface=eth0
VIPNetMask=255.255.255.0
AutoRecover=y
RepBackupDir=/shared_drive/dsbackup
RepBackupPeriod=3600

If you have cache groups in the active standby pair or prefer to recover manually from failure of both master hosts, ensure that AutoRecover is set to n (the default). Manual recovery requires:

  • RepBackupDir specifies a directory on shared storage

  • RepBackupPeriod must be set to a value greater than 0

This example shows attribute settings for manual recovery. The default value for AutoRecover is n, so it is not included in the file.

[manrecoveryDSN]
MasterHosts=host1,host2,host3
ScriptInstallDir=/mycluster/TTscripts
MasterVIP=192.168.1.1, 192.168.1.2
VIPInterface=eth0
VIPNetMask=255.255.255.0
RepBackupDir=/shared_drive/dsbackup
RepBackupPeriod=3600

Using the RepDDL attribute

The RepDDL attribute represents the SQL statement that creates the active standby pair. The RepDDL attribute is optional. You can use it to exclude tables, cache groups and sequences from the active standby pair.

If you include RepDDL in the cluster.oracle.ini file, you do not need to include ReturnServiceAttribute, MasterStoreAttribute and SubscriberStoreAttribute.

When you specify a value for RepDDL, use <DSN> for the data store file name prefix and placeholders for the host names, such as <MASTERHOST[1]> and <MASTERHOST[2]>.

Use the RepDDL attribute to exclude tables, cache groups and sequences from the active standby pair:

[excludeDSN]
MasterHosts=host1,host2,host3,host4
ScriptInstallDir=/mycluster/TTscripts
MasterVIP=192.168.1.1, 192.168.1.2
VIPInterface=eth0
VIPNetMask=255.255.255.0
RepDDL=CREATE ACTIVE STANDBY PAIR \
<DSN> ON <MASTERHOST[1]>, <DSN> ON <MASTERHOST[2]>,\
EXCLUDE TABLE pat.salaries, \
EXCLUDE CACHE GROUP terry.salupdate, \
EXCLUDE SEQUENCE ttuser.empcount

Usually, Oracle Clusterware provides routing information to the TimesTen replication agent. If you are using network interfaces that Oracle Clusterware is not aware of, you can specify them in the ROUTE clause in RepDDL.

Note:

Do not specify a ROUTE clause if you are configuring advanced availability.
[routeDSN]
MasterHosts=host1,host2,host3,host4
ScriptInstallDir=/mycluster/TTscripts
RepDDL=CREATE ACTIVE STANDBY PAIR \
<DSN> ON <MASTERHOST[1]>, <DSN> ON <MASTERHOST[2]>,\
ROUTE MASTER <DSN> ON <MASTERHOST[1]>  SUBSCRIBER <DSN> ON <MASTERHOST[2]>\
MASTERIP "192.168.1.2" PRIORITY 1\
SUBSCRIBERIP "192.168.1.3" PRIORITY 1\ 
MASTERIP "10.0.0.1" PRIORITY 2\
SUBSCRIBERIP "10.0.0.2" PRIORITY 2\
MASTERIP "140.87.11.203" PRIORITY 3\
SUBSCRIBERIP "140.87.11.204" PRIORITY 3\
ROUTE MASTER <DSN> ON <MASTERHOST[2]>  SUBSCRIBER <DSN> ON <MASTERHOST[2]>\
MASTERIP "192.168.1.3" PRIORITY 1\
SUBSCRIBERIP "192.168.1.2" PRIORITY 1\ 
MASTERIP "10.0.0.2" PRIORITY 2\
SUBSCRIBERIP "10.0.0.1" PRIORITY 2\
MASTERIP "140.87.11.204" PRIORITY 3\
SUBSCRIBERIP "140.87.11.203" PRIORITY 3\

Creating and initializing a cluster

To create and initialize a cluster, perform these tasks:

If you plan to have more than one active standby pair in the cluster, see "Including more than one active standby pair in a cluster".

Install Oracle Clusterware

Install Oracle Clusterware. By default the installation occurs on all hosts concurrently. See Oracle Clusterware Administration and Deployment Guide for details.

Oracle Clusterware starts automatically after successful installation.

Install TimesTen on each host

Install TimesTen in the same location on each host in the cluster, including extra hosts.

On Linux and UNIX, the installer prompts you for values for:

  • The T CP/IP port number associated with the TimesTen cluster agent. If you do not provide a port number, TimesTen uses the default TimesTen port.

  • The Oracle Clusterware location.

  • The hosts included in the cluster, including spare hosts, with host names separated by commas

The installer uses these values to create the ttcrsagent.options file.

The ttcrsagent.options file must be the same on every host. On Linux, you can use the -record and -batch options for setup.sh to perform identical installations on additional hosts. See "TimesTen Installation" in Oracle TimesTen In-Memory Database Installation Guide for details.

On Windows, execute ttmodinstall -crs after installation to create the ttcrsagent.options file. You can also use the ttmodinstall utility to create the ttcrsagent.options file on existing installations on other platforms. For more information about ttmodinstall, see "ttmodinstall" in Oracle TimesTen In-Memory Database Reference.

Start the TimesTen cluster agent

Start the TimesTen cluster agent by executing the ttCWAdmin -init command on one of the hosts. For example:

ttCWAdmin -init

When the TimesTen cluster agent has started, Oracle Clusterware begins monitoring the TimesTen daemon and will restart it if it fails.

Note:

You must stop the TimesTen cluster agent on the local host before you execute a ttDaemonAdmin -stop command. Otherwise the cluster agent will restart the daemon.

Create and populate a TimesTen data store on one host

Create a data store on the host where you intend the active data store to reside. The DSN must be the same as the data store file name.

Create schema objects such as tables, AWT cache groups and read-only cache groups.

Create a cluster.oracle.ini file

Create a cluster.oracle.ini file on the host where you intend the active data store to reside. See "The cluster.oracle.ini file" for details about its contents and location.

Create the virtual IP addresses (optional)

For advanced availability, execute the ttCWAdmin -createVIPs command on any host in the cluster. On UNIX, you must execute this command as the root user. For example:

ttCWAdmin -createVIPs -dsn myDSN

Create an active standby pair replication scheme

Create an active standby pair replication scheme by executing the ttCWAdmin -create command on any host. For example:

ttCWAdmin -create -dsn myDSN

This command prompts for an encryption pass phrase that the user will not need again. The command also prompts for the user ID and password for an internal user with the ADMIN privilege if it does not find this information in the sys.odbc.ini file. This internal user will be used to create the active standby pair.

If CacheConnect is enabled, the command prompts for the user password for the Oracle database. The Oracle password is used to set the autorefresh states for cache groups.

Start the active standby pair

Start the active standby pair replication scheme by executing the ttCWAdmin -start command on any host. For example:

ttCWAdmin -start -dsn myDSN

Including more than one active standby pair in a cluster

If you want to use Oracle Clusterware to manage more than one active standby pair in a cluster, include additional configurations in the cluster.oracle.ini file. For example, this cluster.oracle.ini file contains configuration information for two active standby pair replication schemes:

[advancedSubscriberDSN]
MasterHosts=host1,host2,host3
SubscriberHosts=host4, host5
ScriptInstallDir=/mycluster/TTscripts
MasterVIP=192.168.1.1, 192.168.1.2
SubscriberVIP=192.168.1.3
VIPInterface=eth0
VIPNetMask=255.255.255.0

[advSub2DSN]
MasterHosts=host1,host2,host3
SubscriberHosts=host4, host5
ScriptInstallDir=/mycluster/TTscripts
MasterVIP=192.168.1.4, 192.168.1.5
SubscriberVIP=192.168.1.6
VIPInterface=eth0
VIPNetMask=255.255.255.0

Perform these tasks for additional replication schemes:

  1. Create and populate the data stores.

  2. Create the virtual IP addresses. Use the ttCWAdmin -createVIPs command.

  3. Create the active standby pair replication scheme. Use the ttCWAdmin -create command.

  4. Start the active standby pair. Use the ttCWAdmin -start command.

Recovering from failures

Oracle Clusterware can recover automatically from many kinds of failures. This section describes several failure scenarios and how Oracle Clusterware manages the failures.

This section includes these topics:

When an active data store or its host fails

If there is a failure on the node where the active data store resides, Oracle Clusterware automatically changes the state of the standby data store to 'ACTIVE'. If application failover is configured, then the application begins updating the new active data store.

Figure 6-2 shows that the state of the standby data store has changed to 'ACTIVE' and that the application is updating the new active data store.

Figure 6-2 Standby data store becomes active

Description of Figure 6-2 follows
Description of "Figure 6-2 Standby data store becomes active"

Oracle Clusterware tries to restart the data store or host where the failure occurred. If it is successful, then that data store becomes the standby data store.

Figure 6-3 shows a cluster where the former active node becomes the standby node.

Figure 6-3 Standby data store starts on former active host

Description of Figure 6-3 follows
Description of "Figure 6-3 Standby data store starts on former active host"

If the failure of the former active node is permanent and advanced availability is configured, Oracle Clusterware starts a standby data store on one of the extra nodes.

Figure 6-4 shows a cluster in which the standby data store is started on one of the extra nodes.

Figure 6-4 Standby data store starts on extra host

Description of Figure 6-4 follows
Description of "Figure 6-4 Standby data store starts on extra host"

When a standby data store or its host fails

If there is a failure on the standby node, Oracle Clusterware first tries to restart the data store or host. If it cannot restart the standby data store on the same host and advanced availability is configured, Oracle Clusterware starts the standby data store on an extra node.

Figure 6-5 shows the standby data store on a new host after a failure.

Figure 6-5 Standby data store on new host

Description of Figure 6-5 follows
Description of "Figure 6-5 Standby data store on new host"

When read-only subscribers or their hosts fail

If there is a failure on a subscriber node, Oracle Clusterware first tries to restart the data store or host. If it cannot restart the data store on the same host and advanced availability is configured, Oracle Clusterware starts the subscriber data store on an extra node.

When failures occur on both master nodes

This section includes these topics:

Automatic recovery

Oracle Clusterware can achieve automatic recovery from failure on both master nodes if:

  • Advanced availability is configured (virtual IP addresses and at least four hosts)

  • No cache groups are replicated by the active standby pair

  • AutoRecover is set to y

  • RepBackupDir specifies a directory on shared storage

  • RepBackupPeriod is set to a value greater than 0

See "Recovering from failure of both master nodes" for examples of cluster.oracle.ini files.

Manual recovery for advanced availability

This section assumes that the failed master nodes will be recovered to new hosts.

To perform manual recovery in an advanced availability configuration, perform these tasks:

  1. If the new hosts are not already specified by MasterHosts and SubscriberHosts in the cluster.oracle.ini file, then modify the file to include the new hosts. This example uses manrecoveryDSN. This step is not necessary for manrecoveryDSN because extra hosts are already specified in the cluster.oracle.ini file.

  2. Ensure that the TimesTen cluster agent is running on the local host.

    ttCWAdmin -init -hosts localhost
    
  3. Restore the backup data store. Ensure that there is not already a data store on the host with the same DSN as the data store you want to restore.

    ttCWAdmin -restore -dsn manrecoveryDSN
    
  4. If there are cache groups in the data store, drop and re-create the cache groups.

  5. Re-create the active standby pair replication scheme.

    ttCWAdmin -create -dsn manrecoveryDSN
    
  6. Start the active standby pair replication scheme.

    ttCWAdmin -start -dsn manrecoveryDSN
    

Manual recovery for basic availability

This section assumes that the failed master nodes will be recovered to new hosts.

To perform manual recovery in a basic availability configuration, perform these steps:

  1. Acquire new hosts for the data stores in the active standby pair.

  2. Update the MasterHosts and SubscriberHosts entries in the cluster.oracle.ini file. This example uses the basicDSN data store.

  3. Ensure that the TimesTen cluster agent is running on the local host.

    ttCWAdmin -init -hosts localhost
    
  4. Restore the backup data store. Ensure that there is not already a data store on the host with the same DSN as the data store you want to restore.

    ttCWADmin -restore -dsn basicDSN
    
  5. If there are cache groups in the data store, drop and re-create the cache groups.

  6. Re-create the active standby pair replication scheme.

    ttCWAdmin -create -dsn basicDSN
    
  7. Start the active standby pair replication scheme.

    ttCWAdmin -start -dsn basicDSN
    

When more than two master hosts fail

Approach a failure of more than two master hosts as a more extreme case of dual host failure. Use these guidelines:

Planned maintenance

This section includes the following topics:

Changing the schema

Schema changes like altering a table can be performed directly on the active data store. The changes will be replicated to the other data stores in the active standby pair.

If you want to create or drop a table or cache group from an active standby pair in a cluster, perform the following tasks:

  1. Stop the replication agents on the data stores in the active standby pair. These commands use the advancedCacheDSN as an example.

    ttCWAdmin -stop -dsn advancedCacheDSN
    
  2. Drop the active standby pair.

    ttCWAdmin -drop  -dsn advancedCacheDSN
    
  3. Modify the schema as desired.

  4. Re-create the active standby pair replication scheme.

    ttCWAdmin -create -dsn advancedCacheDSN
    
  5. Start the active standby pair replication scheme.

    ttCWAdmin -start -dsn advancedCacheDSN
    

Performing a rolling upgrade of Oracle Clusterware software

See Oracle Clusterware Administration and Deployment Guide.

Upgrading TimesTen

To upgrade TimesTen on one host, perform these tasks:

  1. Stop the replication agents on the data stores in the active standby pair.

    ttCWAdmin -stop -dsn advancedDSN
    
  2. Stop the TimesTen cluster agent on the host. This removes the host from the cluster and stops the TimesTen daemon.

    ttCWAdmin -shutdown -hosts localhost
    
  3. Upgrade TimesTen on the host. If you are upgrading between major releases, use the ttMigrate utility. See "Data Store Upgrades" in Oracle TimesTen In-Memory Database Installation Guide.

  4. Start the TimesTen cluster agent. This includes the host in the cluster and starts the TimesTen daemon.

    ttCWAdmin -init
    

To perform a rolling upgrade of TimesTen across the cluster, perform the tasks on each host.

Adding a read-only subscriber to an active standby pair

To add a read-only subscriber to an active standby pair, perform these steps:

  1. Stop the replication agents on all data stores. This example uses the advancedSubscriberDSN, which already has a subscriber and is configured for advanced availability.

    ttCWAdmin -stop -dsn advancedSubscriberDSN
    
  2. Drop the active standby pair.

    ttCWAdmin -drop -dsn advancedSubscriberDSN
    
  3. Modify the cluster.oracle.ini file.

    • Add the subscriber to the SubscriberHosts attribute.

    • If the cluster is configured for advanced availability, add a virtual IP address to the SubscriberVIP attribute.

    See "Configuring advanced availability" for an example using these attributes

  4. Create the active standby pair replication scheme.

    ttCWAdmin -create -dsn advancedSubscriberDSN
    
  5. Start the active standby pair replication scheme.

    ttCWAdmin -start -dsn advancedSubscriberDSN
    

Removing a read-only subscriber from an active standby pair

To remove a read-only subscriber from an active standby pair, perform these steps:

  1. Stop the replication agents on all data stores. This example uses the advancedSubscriberDSN, which has a subscriber and is configured for advanced availability.

    ttCWAdmin -stop -dsn advancedSubscriberDSN
    
  2. Drop the active standby pair.

    ttCWAdmin -drop -dsn advancedSubscriberDSN
    
  3. Modify the cluster.oracle.ini file.

    • Remove the subscriber from the SubscriberHosts attribute or remove the attribute altogether if there are no subscribers left in the active standby pair.

    • Remove a virtual IP the SubscriberVIP attribute or remove the attribute altogether if there are no subscribers left in the active standby pair.

  4. Create the active standby pair replication scheme.

    ttCWAdmin -create -dsn advancedSubscriberDSN
    
  5. Start the active standby pair replication scheme.

    ttCWAdmin -start -dsn advancedSubscriberDSN
    

Adding an active standby pair to a cluster

To add an active standby pair (with or without subscribers) to a cluster that is already managing an active standby pair, perform these tasks:

  1. Create and populate a data store on the host where you intend the active data store to reside initially. See "Create and populate a TimesTen data store on one host".

  2. Modify the cluster.oracle.ini file. This example adds advSub2DSN to the cluster.oracle.ini file that already contains the configuration for advancedSubscriberDSN. The new active standby pair is on different hosts from the original active standby pair.

    [advancedSubscriberDSN]
    MasterHosts=host1,host2,host3
    SubscriberHosts=host4, host5
    ScriptInstallDir=/mycluster/TTscripts
    MasterVIP=192.168.1.1, 192.168.1.2
    SubscriberVIP=192.168.1.3
    VIPInterface=eth0
    VIPNetMask=255.255.255.0
    
    [advSub2DSN]
    MasterHosts=host6,host7,host8
    SubscriberHosts=host9, host10
    ScriptInstallDir=/mycluster/TTscripts
    MasterVIP=192.168.1.4, 192.168.1.5
    SubscriberVIP=192.168.1.6
    VIPInterface=eth0
    VIPNetMask=255.255.255.0
    
  3. Create new virtual IP addresses. On Linux and UNIX, the user must be root to do this.

    ttCWAdmin -createVIPs -dsn advSub2DSN
    
  4. Create the new active standby pair replication scheme.

    ttCWAdmin -create -dsn advSub2DSN
    
  5. Start the new active standby pair replication scheme.

    ttCWAdmin -start -dsn advSub2DSN
    

Removing an active standby pair from a cluster

To remove an active standby pair (with or without subscribers) from a cluster, perform these tasks:

  1. Stop the replication agents on all data stores in the active standby pair. This example uses advSub2DSN, which was added in "Adding an active standby pair to a cluster".

    ttCWAdmin -stop -dsn advSub2DSN
    
  2. Drop the active standby replication scheme.

    ttCWAdmin -drop -dsn advSub2DSN
    
  3. Drop the virtual IP addresses for the active standby pair.

    ttCWAdmin -dropVIPs -dsn advSub2DSN
    
  4. Modify the cluster.oracle.ini file (optional). Remove the entries for advSub2DSN.

  5. If you want to destroy the data stores, log onto each host that was included in the configuration for this active standby pair and use the ttDestroy utility.

    ttDestroy advSub2DSN
    

    For more information about ttDestroy, see "ttDestroy" in Oracle TimesTen In-Memory Database Reference.

Adding a host to the cluster

Adding a host requires that the cluster be configured for advanced availability.

To add a host to the cluster, perform these tasks:

  1. Stop the replication agents. This example uses the example DSN advancedDSN.

    ttCWAdmin -stop -dsn advancedDSN
    
  2. Drop the active standby pair.

    ttCWAdmin -drop -dsn advancedDSN
    
  3. Modify the cluster.oracle.ini file. In this example, host5 is added to the list of master hosts.

    [advancedDSN]
    MasterHosts=host1, host2, host3, host4, host5
    ScriptInstallDir=/mycluster/TTscripts
    MasterVIP=192.168.1.1, 192.168.1.2
    VIPInterface=eth0
    VIPNetMask=255.255.255.0
    
  4. Ensure that the TimesTen cluster agent is running on host5.

    ttCWAdmin -init -hosts host5
    
  5. Re-create the active standby pair replication scheme.

    ttCWAdmin -create -dsn advancedDSN
    
  6. Start the active standby pair replication scheme.

    ttCWAdmin -start -dsn advancedDSN
    

Removing a host from the cluster

Removing a host from the cluster requires that the cluster be configured for advanced availability. MasterHosts must list more than two hosts if one of the master hosts is to be removed. SubscriberHosts must list at least one more host than the number of subscriber data stores if one of the subscriber hosts is to be removed.

To remove a host from the cluster, perform these tasks:

  1. Stop the replication agents. This example uses the example DSN advancedSubscriberDSN.

    ttCWAdmin -stop -dsn advancedSubscriberDSN
    
  2. Drop the active standby pair.

    ttCWAdmin -drop -dsn advancedSubscriberDSN
    
  3. Remove the host from the cluster. See "Adding and Deleting Cluster Nodes" in Oracle Clusterware Administration and Deployment Guide for details.

  4. Modify the cluster.oracle.ini file by removing the host from the appropriate list of hosts. In this example, host5 has been removed from the list of subscriber hosts.

    [advancedSubscriberDSN]
    MasterHosts=host1,host2,host3
    SubscriberHosts=host4
    ScriptInstallDir=/mycluster/TTscripts
    MasterVIP=192.168.1.1, 192.168.1.2
    SubscriberVIP=192.168.1.3
    VIPInterface=eth0
    VIPNetMask=255.255.255.0
    
  5. Re-create the active standby pair replication scheme.

    ttCWAdmin -create -dsn advancedSubscriberDSN
    
  6. Start the active standby pair replication scheme.

    ttCWAdmin -start -dsn advancedSubscriberDSN
    

Reversing the roles of the master data stores

After a failover, the active and standby data stores are on different hosts than they were before the failover. Use the -switch option of the ttCWAdmin utility to reverse the roles of the active and standby data stores:

ttCWAdmin -switch -dsn basicDSN

Moving a data store to a different host

When a cluster is configured for advanced availability, you can use the -relocate option of the ttCWAdmin utility to move a data store from the local host to the next available spare host specified in the MasterHosts attribute in the cluster.oracle.ini file. If the data store on the local host has the active role, the -relocate option first reverses the roles so that the standby store in the same cluster becomes active. The newly migrated data store on the extra host always comes up as the standby data store. This option is useful for relocating a data store if you need to take the host offline. For example:

ttCWAdmin -relocate -dsn advancedDSN

Performing host or network maintenance

If you need to upgrade the operating system or hardware for a host or perform network maintenance, shut down Oracle Clusterware and disable automatic startup. Execute these Oracle Clusterware commands as root or OS administrator:

# crsctl stop crs

# crsctl disable crs

Shut down TimesTen. See "Shutting down a TimesTen application" in Oracle TimesTen In-Memory Database Operations Guide.

Perform the host maintenance. Then enable automatic startup and start Oracle Clusterware:

# crsctl enable crs

# crsctl start crs

See Oracle Clusterware Administration and Deployment Guide for more information about these commands.

Performing maintenance on the entire cluster

When all of the hosts in the cluster need to be brought down, stop Oracle Clusterware on each host individually. Execute these Oracle Clusterware commands as root or OS administrator:

# crsctl stop crs

# crsctl disable crs

Shut down TimesTen. See "Shutting down a TimesTen application" in Oracle TimesTen In-Memory Database Operations Guide

Perform the maintenance. Then enable automatic startup and start Oracle Clusterware:

# crsctl enable crs

# crsctl start crs

Changing user names or passwords

When you create the active standby pair replication scheme with the ttCWAdmin -create command, Oracle Clusterware prompts for the user name and password of the internal user. If there are cache groups in the active standby pair, Oracle Clusterware also stores the cache administration user name and password. To change the user name or password for the internal user or the cache administration user, you must re-create the cluster.

To change the user name or password of the internal user that created the active standby pair replication or to change the cache administration user name or password, perform these tasks:

  1. Stop the replication agents on the data stores in the active standby pair. These commands use the advancedCacheDSN as an example.

    ttCWAdmin -stop -dsn advancedCacheDSN
    
  2. Drop the active standby pair.

    ttCWAdmin -drop -dns advancedCacheDSN
    
  3. Change the appropriate user name or password:

  4. Re-create the active standby pair replication scheme.

    ttCWAdmin -create -dsn advancedCacheDSN
    
  5. Start the active standby pair replication scheme.

    ttCWAdmin -start -dsn advancedCacheDSN