05: Redundant Architectures

CAMIO Cluster

CAMIO Cluster provides users with a high availability asset management/MOS solution with fully redundant servers. Automatic and immediate failover is initiated in the event of a hardware failure. CAMIO Cluster is based on Windows Failover Cluster Server with SQL Server. It provides a high reliability RAID assembly for media storage and full hardware redundancy.

Pros vs. Cons

Pros

• Fully automated, very fast failover with no user intervention required.
• Since primary and backup share the same RAID storage, the data is always up to date after a
• failover to the backup system.

Cons

• Entire system is co-located so this is not a true disaster recovery solution.
• The system does not automatically make backups of media files. Users should make regular
• backups to recover from potential data corruption cases.

Newsroom clients and playout systems do not require any changes during a failover. When properly configured with the user’s SMTP mail server, CAMIO Cluster can send an email notification of failure events.

CAMIO Cluster Server or VM Specifications

CAMIO Cluster Server requires two CAMIO physical or VM servers configured as a MSSQL SQL Cluster with a common virtual IP, and a shared NAS/SAN Storage. In the case of VMs, it is recommended that the CAMIO VM server pairs are installed on separate VM hosts for redundancy.

 

• After Failover due to an issue on the Primary CAMIO: If the Primary CAMIO Node is unreachable or having some failure at the infrastructure or application level, Microsoft Failover Cluster operates an automatic failover to the Backup CAMIO Node by assigning the Virtual IP Address and starting the SQL Server and CAMIO Services.

Failback procedure

After recovery of CAMIO Node1, the Failback procedure, if configured, can be automated and run immediately or at specific time slots. However it has some risks and is preferred to be manual. To be executed during downtime hours to avoid impacting the production. To proceed,

1. Connect to the CAMIO Node 2 (or Node 1 if correctly recovered) and launch the Failover Cluster Manager application
2. From the left panel, go to Nodes and check that CAMIO Node 1 is up and healthy

   

1. Once validated, go to Roles. There should be one unique Role, responsible for managing all clustered Resources: the Cluster virtual IP address and Network Name, the Storage partitions used by SQL Server and CAMIO, and also the SQL Server and CAMIO Services:

   

    

2. Right-click on the Role and select Move > Select Node…
3. Note: do not run this operation on specific Resources of that Role (CAMIO Services, Storage File Server, SQL) as it may affect the whole system badly.

   

4. Then select the CAMIO Node 1

CAMIO DR

CAMIO DR is a new option for users who want to protect their data, but at an overall lower price. CAMIO DR is a warm standby backup system specifically designed for CAMIO servers. It performs regularly scheduled backups of the primary CAMIO’s assets and databases.

In the event there is a failure of the primary system, a manual procedure is used to fail-over operation to the CAMIO DR system, restoring the system to the state of the last backup. After repairing or replacing the primary system, a manual procedure is used to fail-back operation to the CAMIO primary system.

Pros and Cons

Pros

• Lower cost, especially if a CAMIO is already deployed at the user’s site.
• The backup system can be remotely located (assuming fast LAN connection) for disaster recovery scenarios.
• System makes scheduled data backups, which could be restored in case of data corruption.

Cons

• Data is backed up on a schedule. After a failover, data will only be as current as the last backup.
• The failover procedure is manual, hence no high availability.

Redundancy Capabilities

• During normal operation: Database backups are periodically made on the Primary CAMIO Server and copied along with the content of the D, I and J Drives onto the Backup CAMIO DR Server, approximately every 5 minutes.
  • The Primary CAMIO Server’s NIC is configured with an extra Virtual IP Address that is used as CAMIO main endpoint. The CAMIO Windows Services on the Primary Server are running while stopped on the Backup Server.

• After Failover due to an issue on the Primary CAMIO: If the Primary CAMIO Server is unreachable or having some failure at the infrastructure or application level, an Administrator can run the Failover procedure from the Backup CAMIO DR using the DR Client application. It will perform the following actions sequentially: restore the database using the last copied backup file, configure the Virtual IP Address on the NIC and start the CAMIO Windows Services.

Failover Procedure

To execute the failover procedure in order to activate the Backup CAMIO DR Server, the Administrator should proceed with the following steps:

Connect to the Backup CAMIO DR Server and launch the DR Client application:

The Machine Status tab should confirm that the Primary CAMIO Server is unreachable (Remote Machine Mode = Offline):

From the Data Recovery Status tab, run the Failover Primary to Secondary without Final Backup option:

This will perform the following actions:

• Checking again that the Primary CAMIO server is unreachable
• Enabling the Virtual IP Address by adding it into the main NIC Settings
• Restoring the last copied database backup
• Starting the IIS Web Server
• Starting all the CAMIO Services

The following messages are displayed when complete:

Failback Procedure

After recovery of the Primary CAMIO Server, the Administrator can run the Failback procedure as follow:

Connect to the Backup CAMIO DR Server and launch the DR Client application:

The Machine Status tab should confirm that the Primary CAMIO Server is back and inactive (Remote Machine Mode = Passive):

From the Data Recovery Status tab, run the Fail back from Secondary to Primary option:

This will perform the following actions:

• Copying the last database backup and files to the Primary server
• Stopping the IIS Web Server
• Stopping all the CAMIO Services
• Backuping the Database and copying it along with any newly updated file to the Primary server since the 1st copy
• Disabling the Virtual IP Address
• Enabling the Virtual IP Address on the Primary server by adding it into the main NIC Settings
• Restoring the last copied database backup on the Primary server
• Starting the IIS Web Server on the Primary server
• Starting all the CAMIO Services on the Primary server

The following messages are displayed when complete: