Crash Protection Settings

Unattended Monitoring via Crash Protection is a way to protect your server from various types of load. You can protect from excessive numbers of requests all being run at the same time. You can protect from individual requests running for long periods of time and you can protect from requests running when the server is low on memory. FusionReactor can notify you when any of these situations occur.

 

The Crash Protection Settings page lets you modify various elements relating to the different forms of Crash Protection.

 

How do I set up Unattended Monitoring with Crash Protection?

From the FusionReactor Administrator, click "CP Settings" from the table of contents (Crash Protection section) and you will see the Crash Protection Settings screen. From here you can activate any of the crash protection triggers and set up what action should be taken if they trigger:

 

Crash Protection:

Timeout Protection (s)

By entering a number (in seconds) in this field you will enable Timeout Crash Protection, from that point on, any request which takes longer to execute than this time will trigger Crash Protection. To turn this feature off again, simply remove any value from the field. You can specify what should occur when Timeout Crash Protection is triggered:

  • Abort and Notify - The request will be aborted and a notification email will be sent

  • Notify - The request is left running and a warning email is sent.

Memory Protection (%)

By entering a number (percentage) in this field you will enable Memory Crash Protection. From that point on, if the amount of free memory on the server is less than this number when a request arrives, Crash Protection will be triggered. To turn this feature off again, simply remove any value from the field. You can specify what should occur when Memory Crash Protection is triggered:

  • Abort and Notify - The request will be aborted and a notification email will be sent

  • Queue and Notify - The request will be added to a queue in order to be executed once the server has recovered and a notification email will be sent. You can specify the queue timeout in the Survival Strategy section of this page.

  • Notify - The request is left running and a warning email is sent.

Request Protection

By entering a number in this field you will enable Request Quantity Crash Protection. From that point on, if the amount of currently running requests on the server is greater than this number when a request arrives, Crash Protection will be triggered. To turn this feature off again, simply remove any value from the field. You can specify what should occur when Request Quantity Crash Protection is triggered:

  • Abort and Notify - The request will be aborted and a notification email will be sent

  • Queue and Notify - The request will be added to a queue in order to be executed once the server has recovered and a notification email will be sent. You can specify the queue timeout in the Survival Strategy section of this page.

  • Notify - The request is left running and a warning email is sent.

 

NOTE: In order to receive notification email you must set up your Email Server on the FusionReactor Settings page. You can also change the behaviour of the Request Abort and Request Queue in the Survival Strategy part of this page.

 

Crash Protection Restrictions:

Restrictions

From here you can enable or disable the Crash Protection Restrictions feature. When restrictions are enabled you can define which pages will be run through the Crash Protection system and which will be left to run normally. You can defined individual restrictions on the CP Restrictions page.

Behavior

This value defines how Crash Protection Restrictions behave.

  • Protect Matching Requests - No requests will be subject to protection (they will never be aborted or timed out) unless they are matched by a Restriction rule as defined on the CP Restrictions page.

  • Exclude Ignore Requests -  All requests will be subject to protection (they may be aborted or timed out) unless they are matched by a Restriction rule, in which case they will not be touched by Crash Protection under any circumstances as defined on the CP Restrictions page.

 

 

What is a Survival Strategy and Self-healing?

The survival strategy is how FusionReactor will attempt to self-heal the situation if crash protection is triggered. It can be configured to kill problem requests and queue new requests until the situation is stabilized. It will also alert you so that you have time to resolve the problem if you need to take action.

 

 

Survival Strategy:

Abort Strategy

When a request is aborted due to Crash Protection there are two possible behaviours:

  • Display Message - Enter your message in the text box provided and that message will be presented to the user who made the aborted request.

  • Redirect to URL - Enter a URL in the text box provided and the user who made the aborted request will instead be redirected to this URL.

NOTE: Redirects may not work when a page has already sent data to the browser.

Add Parameters

After an abort, if you "Redirect to URL" then this option defines whether FusionReactor will add extra parameters to your URL so you can see exactly what happened. If you enable parameters the your URL will include the following:

  • detection_method - This is the type of Crash Protection which fired. It will be one of: "freemem", "requests", or "timeout"

  • threshold_value - This is the threshold value associated with the type of Crash Protection, as defined earlier on this page.

  • actual_value - This is the actual which caused the Crash Protection to fire.

  • source_url - Contains the URL sent on the original request e.g. http://127.0.0.1/test.cfm  (since 3.0.1)

  • source_url_params - Contains the URL parameters sent on the original request (since 3.0.1)

  • source_method - This is the HTTP method used on the original request e.g. GET, POST (since 3.0.1)

  • start_time - This is the start time of the request from Epoch e.g. 1209376581900 (since 3.0.1)

Queue Timeout / (Request Abort) (seconds)

If a request is queued due to Crash Protection, this value represents the longest amount of time that the request will be held before FusionReactor abandones it.

 

Crash Protection Logging:

CP Logging

If Crash Protection Logging is enabled then Crash Protection information will be written to the Crash Protection Log.

View Size (KB)

This value indicates how much of the log file should be displayed when you view the Crash Protection Log from inside FusionReactor.

File Size (KB)

Specifies the size of each file in the Crash Protection Log rotating file set.

File Count

Specifies the number of files in the Crash Protection Log rotating file set.

 

How do I set up Notifications?

From the FusionReactor Administrator, click "Settings" from the table of contents (FusionReactor section) and you will see the FusionReactor Settings screen. The Email section of this screen deals with the notification.

 

Why would I want to enable Crash Protection?

There are many cases in which crash protection can be an invaluable tool. These include:

Easing Support

Crash Protection automates the manual process of checking each of your servers and verifying that they are operating within satisfactory parameters. Instead of having to periodically go through each server you can have warnings sent directly to your inbox.

Request Bursts

If you have a server which receives bursts of requests all at once then FusionReactor can using queuing spread that load out over time and so protect your server from becoming overwhelmed.

Preventing Crashes

If you periodically have very resource hungry jobs running then FusionReactor can stop new requests being run which could potentially cause the server to become unstable.

During Development

If you need to write a page which cannot use more that a fixed amount of the system resources then crash protection is a good way of tracking where and when you have problems. As well as emails, crash triggers can also be viewed in a logfile.