Skip to content

Configuring SAS for ESM

Having successfully installed the ESM Server and Agent components, SAS environment configuration is now required to enable ESM to monitor the various components of your SAS installation.

Note

It is highly recommended that you read and understand the document titled Functional Overview of ESM before continuing with the tasks outlined here. This document is intended to explain the concepts involved in SAS configuration for ESM on both Windows and UNIX based platforms and provides examples for both.

SAS Server Configuration

This example shows what a SAS configuration directory structure for an environment with 2 levels may look like (individual files are omitted):

Config
   Lev1
     SASApp
       WorkspaceServer
       PooledWorkspaceServer
       StoredProcessServer
       BatchServer
       ConnectServer
     SASBatch
       WorkspaceServer
       PooledWorkspaceServer
       ConnectServer
     SASLASR
     SASEnterpriseMiner
   Lev2
     SASApp
     SASBatch

LevN subdirectories contain configuration files used by SAS to instantiate the various component processes for a given SAS configuration. The contents of each subdirectory vary by component. Multiple Levels (Levs) are often used to enable a physical host to be logically divided to support development, testing and production. For example, Lev1 may be used for Production and Lev2 and Lev3 for Testing and Development, respectively.

If a node is split logically into multiple Levels, it is important that all active components are configured for monitoring with ESM regardless of Lev. If this is not done, certain sessions on the server will be invisible to ESM, and will not be considered when analysing the performance of the node they are executing on.

Changes to Server Context Configuration Files

The following is an example of typical User Modifiable configuration files found in the root directory of a SAS Application Server Context definition.

Config
   Lev1
     SASApp
       WorkspaceServer
         appserver_autoexec_usermods.sas
         appservercontext_env_usermods.sh

At the Application Server level, changes will be made to these files so that they do the following:

appservercontext_env_usermods

  • Set the ESM Installation directory environment variable
  • Include and source the esmconfig script from the ESM Agent config directory.
  • Set the ESMACTIVE flag. This setting determines whether the components inside this this Application Server Context that are configured for monitoring with ESM, actively report in to ESM. Default setting is 1 (enabled)

Typically, the code added to this file looks something like this:

# ESM CONFIG START

export ESMBASEDIR=/opt/esm/esm-agent
# source the script
. $ESMBASEDIR/conf/esmconfig.sh
# set the active flag as 1 for active, 0 for inactive
export ESMACTIVE=1
# NOTE: The ESMACTIVE flag is set independently in the OLAPServer config

# This is the name given to non-job sessions
export ESMSESSIONNAME=Lev1_SASApp

# ESM CONFIG END
rem ESM CONFIG START 

Set ESMBASEDIR=c:\esm\esm-agent 
rem source the script 
call %ESMBASEDIR%\conf\esmconfig.bat 
rem set the active flag: 1 for active, 0 for inactive 
Set ESMACTIVE=1 
rem NOTE: The ESMACTIVE flag is set independently in the OLAPServer config 

rem This is the name given to non-job sessions 
Set ESMSESSIONNAME=Lev1_SASApp 

rem ESM CONFIG END

appserver_autoexec_usermods.sas

  • Read the ESM Installation directory from the environment variable configured above
  • Include the esmtags.sas program from the ESM Agent sasautos directory

Typically, the code added to this file looks something like this:

/* ESM CONFIG START */ 

%let ESMBASEDIR=%SYSGET(ESMHOME);
%include "&ESMBASEDIR/sasautos/esmtags.sas";

/* ESM CONFIG END */

Changes to individual Server Config files

Within each Logical Server subdirectory are the configuration files for each server. Files in each of these will require some additions in order for the Servers to be picked up and monitored by ESM at runtime. The files that need to be modified for the most common SAS Server types are outlined in the following table:

autoexec_usermods Server Server_usermods Metadata config
Workspace
Pooled Workspace
Batch
Stored Process
Connect
OLAP

- update required to file - update required on Windows platform only
Server is invocation script for that server (i.e. sasbatch.sh or WorkspaceServer.sh)

Server_usermods script changes

In order to associate SAS sessions to their parent server types, an environment variable is set in the Server_usermods file:

# ESM CONFIG START

export ESMSESSIONTYPE=OLAP

# ESM CONFIG END

This is the only change that needs to be made for most SAS Server types.

Invocation Script Changes

Datastep Batch Servers (and Stored Process Servers on Windows Platforms) require their main invocation scripts to be modified to allow the ESM Agent to capture enhanced information, such as batch job return codes and Multibridge session port numbers respectively.

Typical configuration changes to the SAS DATA Step Batch Server Run File look like this:

# ESM CONFIG START

# The following should be pasted immediately after the line beginning with 
# exec "$SAS_COMMAND", once the exec command has been removed.
rc=$?
$ESMHOME/reportOut.sh $ESMNODENAME $ESMGUID $ESMJOBNAME $rc

# ESM CONFIG END
exit $rc
REM ESM CONFIG START

REM Paste this snippet immediately after the set rc=%ERRORLEVEL% line
call "%ESMHOME%\reportOut.bat" %ESMNODENAME% %ESMGUID% "%ESMJOBNAME%" %rc%

REM ESM CONFIG END
exit %rc%

Warning

Incorrect or incomplete modification of the sasbatch.sh or sasbatch.bat script can affect how schedulers such as Platform LSF interpret the status of finished jobs, which can in turn affect the flow and triggering of subsequent jobs.

When modifying these scripts ensure that the rc=$? or set rc=%ERRORLEVEL% commands continue to capture the return code of the job launched by the SAS command, and that the sasbatch.sh or sasbatch.bat script itself exits returning the same return code.

Metadata Configuration Changes

Stored Process Server requests are executed by a pool of Multibridge sessions. Multibridge sessions are runner processes often pre-initialised by the SAS Object Spawner which execute Stored Process program code when requested to do so by the user. In order to capture the attributes of each request into an Event so that ESM can identify the program and person currently using that multibridge session, a Request Initiation program must be specified in metadata. This is done using SAS Management Console.

The SAS Stored Process Server Metadata configuration:
STP Server Metadata Configuration

The Boemska ESM Configuration Assistant

To assist with the configuration changes required for ESM integration, Boemska provide a utility called the ESM Configuration Assistant. The Configuration Assistant is designed to automate as much of the above configuration procedure as possible, minimising the chance of error while leaving ultimate control of the changes with the user making them.

In Define Mode, the utility requires input from a SAS Administrator to define the SAS environment and each of the components to be configured for monitoring. It uses the inputs to generate the necessary changes required to configure the various scripts, autoexecs and configuration files used by your SAS installation, and then in Configure Mode guides the Administrator making the changes by asking them to copy and paste the generated commands and code into a terminal. By doing this the changes can be reviewed before they are committed and the chance of user error is minimised.

The ESM Configuration utility generates a downloadable Backup Script which can be used to validate that the files specified in Define Mode exist on the filesystem, and makes a timestamped copy of those files prior to any changes being made. After Configure Mode is completed and the changes are made, the utility also generates a Deconfiguration Script, which parses for ESM CONFIG START and ESM CONFIG END comment blocks in any files marked as changed, and removes any changes between them. This automates the removal of the configuration changes that were required for the integration, should ESM ever need to be removed from the installation.

As mentioned above, the ESM Configuration Assistant workflow is made up of two stages:

  • Define Mode lets the user define the Levels, Server Contexts and Application Servers for each of their configuration components. The assistant is pre-configured with the default settings for each component and their respective configuration files. If a SAS installation differs from the defaults, the Configuration Assistant lets the user override the suggested settings to ensure the Configuration Assistant generates the correct changes to the correct files.

  • In Configure Mode the Configuration Assistant generates the changes required to each component and guides the user through each change. Select Text buttons automatically highlight the required text for easy copying to the clipboard. These changes can then be pasted in to the relevant file on the SAS Server. If any of the copied commands fail to open the correct files, return to Define Mode for that component and update the definitions so that the correct files are referenced. This will ensure that any changes made to the configuration are traceable and correctly referenced by the deconfiguration scripts.

Click here to open the ESM Configuration Assistant application.

Define Mode

When first opened, the ESM Configuration Assistant will automatically launch in Define Mode.

ESM Configuration Assistant main

Before continuing with the rest of the configuration, some basic parameters must be set, found under the Config component in the configuration tree. These are as follows:

System Type

UNIX (for Unix and Linux based systems) or WIN (for Windows based systems). This is the operating system of the application server to be monitored. For environments where application server clusters contain a mix of Windows and UNIX based Operating Systems, two separate instances of the Assistant application must be used for updating the configurations.

Preferred Editor

Either choose your preferred text editor from a predefined list, or enter your own. This is used to build the command which will be used to open the required configuration files in the editor. If your preferred editor is not accessible via the system path it may need to be defined by its full path. For example, on a Windows environment, the full path for the Notepad++ editor executable may be defined as:

"C:\ Program Files (x86)\ Notepad++\ notepad++.exe"
To add an editor manually, click or tab into the drop down box and edit the text, clicking on the update button to save the value.

Config Root

This should point to the filesystem location where the SAS configuration resides. Exclude the LevN directory from this path. An example of this value on a UNIX based system:

/opt/sas94/config

ESM Root

This should point to the filesystem location where the ESM Agent is installed. This value maps to the agent install path in the installation wizard, so on a UNIX based system where the agent is installed in /opt/esm/esm-agent, this value would be entered as:

/opt/esm/

At this point, the Administrator can replicate their target environment configuration by adding the LevN and Server components to the Component Tree on the left hand side of the Assistant application. A subcomponent is added by selecting its parent node in the component tree, and then clicking the Add (context aware) Component button. The parameters are then saved to that node by clicking the Update button on that node's properties panel.

The following GIF animation demonstrates the Define Mode workflow:

ESM Define GIF

After all Server Contexts, Application Servers and Levels have been defined in Define Mode, the neccessary files need to be updated. This is done using Configure Mode.

Configure Mode

The Configure Mode of the ESM Configuration Assistant guides the user through the following process:

  • Loading each of the files to be changed into the Preferred Editor
  • Making the change specific to that file
  • Recording an audit trail of all files changed as part of the configuration process

Configure Mode is started by clicking the Configure Mode button in the top left corner of the screen.

In Configure Mode, the Previous Component and Next Component buttons are used to step through each of the components defined in the configuration tree. The panel on the right, in each case, contains instructions specific to the currently active component.

The majority of instructions involve adding snippets of text to various configuration files. Where a defined file needs to be opened in the Editor, the command to do so is automatically generated by the Assistant, to minimise possibility of any typing errors. The Select Text buttons next to the command should also be used to avoid selection errors, and to speed up the process of copying the command to the clipboard. The general Configure Mode workflow is as follows:

  • Open the component to be configured using the Next Component button
  • Read the instructions specific to that component
  • Where there is a file that needs updating:
    • Click the Select Text button to select the command to open the file
    • Copy the command to the clipboard
    • Paste it into the terminal
    • Review, hit Enter
    • Click the Select Text button to select the snippet of text to be added to the file
    • Copy the snippet to the clipboard
    • Paste it into your editor
    • Review the change to the file and save it
  • For every change that is made, tick the I have made these changes tickbox in the configurator app.

The following GIF animation demonstrates this Configure Mode workflow:

ESM Configure GIF

Once all of the required configuration changes have been made, it is neccessary to restart the Object spawners, Connect Spawners and OLAP Servers in order for the changes to the Server Context configuration to come into effect. Once this has been done, your environment should be fully integrated with ESM. However, it is always a good idea to validate each of the components prior to resuming the normal operation of your environment.

Validating the Changes

The configuration changes for each Server can be validated as follows:

Workspace Server

This can be done either using SAS Management console, or using a client such as SAS Enterprise Guide.

To do this via SAS Management Console, navigate to the Server Manager plugin, and locate the SAS Workspace Server under the context that has been configured (eg. SASApp). Right click, and select the Validate option.

Management Console server validation

The following two things should happen as a result:

  1. You should see the following 'Validation Successful' message
    Management Console successful validation

  2. A few seconds later, a session should appear in the List Portlet of the ESM Live View:
    ESM Config Workspace Validation

Note

When SAS Management Console is used to validate the Workspace Server, the session that it spawns is terminated immediately following validation. It is therefore normal for the session to be greyed out as soon as it appears in the ESM Live View. Standard Workspace Sessions started by a client application should persist as expected.*

Alternatively, to validate the Workspace Server via a client such as SAS Enterprise Guide, simply connect to the Server Context in question and execute some SAS code. Your session should appear in the ESM Live View, and should be correctly associated to your SAS Metadata username.

Connect Server

The procedure for validating the SAS/CONNECT Server is very similar to that of a standard SAS Workspace Server. However, instead of right-clicking on the Logical Server definition in SAS Management Console, right-click the server definition itself.

Management Console server validation

SAS/CONNECT sessions should also be validated by connecting to the SAS/CONNECT server using a suitable client.

SAS OLAP Server

The procedure for validating the SAS OLAP Server configuration uses the SAS Management Console based Validate command, similarly to the validation for the SAS/CONNECT Server or SAS Workspace Server.

SAS Stored Process Server

The Stored Process Server configuration can also be validated from SAS Management Console in the same way as the Workspace Server configuration. However, if the Start Size in the Load Balancer configuration is set to a number higher than zero, the Multibridge Sessions will be started by the spawner, making the Management Console based validation unneccessary.

In either case, there should be at least one Multibridge Session visible in the Live View list portlet:

STP Session Live Validation

Note that the Multibridge Port (8611 in this example) should be identified correctly.

To validate the addition to the Metadata configuration which allows each individual request to be identified, execute a stored process using a supported client (such as the SAS Stored Process Web Application). Depending on the Load Balancing configuration, a flag should be drawn against one of the Multibridge Session portlets.

STP Session Live Validation

Hovering over the flag should display a tooltip correctly identifying the test program executed.

Pooled Workspace Server

The Pooled Workspace Server configuration should be validated in the same way as the Stored Process Server configuration. However, to issue a request to a Pooled Workspace session in order to validate the identification of that request, you will need to execute an Information Map against that session. This can be done via a Pooled Workspace client application such as SAS Information Map Studio or SAS Web Report Studio.

Pooled Workspace Session Live Validation

SAS Batch Server

To help validate the extended functionality for monitoring SAS Batch Jobs, ESM ships with a test Job which is to be used when validating the configuration of the SAS Batch Server. To trigger the batch job, navigate to a directory where you have write permissions, and invoke the sasbatch.sh / sasbatch.bat script, passing it the ESM Validation Job as input. Here is an example using the Windows platform:

C:\> c:\sas\Config\Lev1\SASApp\BatchServer\sasbatch.bat -sysin c:\bts\esm-agent\sasautos\validateBatch.sas

The following should happen:

  1. The Batch Job should appear among the other active sessions in the Live View Session List portlet, showing validateBatch.sas in the Name column. Batch Server Session Live Validation

  2. Once finished, the Session Portlet for the job should look similar to this: Batch Server Session Portlet Validation It should exhibit the following features:

    • The job should be roughly bound by a light blue background, signifying it is a job.
    • The background area should be titled with the name of the job across the top.
    • If SASWORK monitoring is permitted to the agent at the Operating System level, blue bars should show the SASWORK tracking working correctly.
    • A white tag titled 'User Tag' should be showing halfway through the job.
    • An ERROR and a WARNING flag should be apparent, highlighting the (artificial) error and warning in the job log.
  3. Clicking on one of the E or W flags should retrieve the Job Log from the node's filesystem (if permitted by OS) Batch Server Session Log Retrieval

  4. Typing 'validate' in the Job Name box and clicking the Graph button should load a graph showing the execution of the ValidateBatch.sas job, signified with a green bar.

Batch Server Session Job Status

If you are unable to see any of the above, please contact Boemska Suport at support@boemskats.com.

This concludes the Validation step of ESM Configuration.

Exceptions and troubleshooting

SAS OLAP Server for Windows platforms

The SAS OLAP Server on the Windows Platform is unable to source the variables defined in esmconfig.bat, as it does not source the appservercontext_env_usermods script. As a workaround, the ESMACTIVE macro variable value is set manually in the OLAP Server autoexec_usermods.sas file. If you are intending to use the ESMACTIVE environment variable to turn off ESM monitoring for a Windows-based environment, you will also need to independently change the ESMACTIVE variable in the OLAP Server's local autoexec_usermods.sas configuration file.

Pooled Workspace or Stored Process Servers TK Kernel Errors

These errors can occur when the incorrect path to a Request Initialisation or Session Initialisation program is configured in SAS Metadata, or if the sassrv process can not read the file. If this error appears, check the .sas files configured in SAS Metadata exist and are readable by the sassrv user. More information about the setting in question can be found under the Metadata Configuration Changes section of this document.

Batch Server Logfile Tracking and logconfig.xml configuration

Metadata configured logfile pattern
Logfile pattern configured under Metadata

If your SAS Data Step Batch Server configuration uses the server context logconfig.xml file to specify the logfile filename pattern rather than having it specified in the Batch Server Metadata definition (i.e. when your environment is configured for APM), some additional steps will be required in order to fully integrate the Batch Server with the logfile tracking functionality of ESM.

The ESM integration macros rely on the SAS System Options to ascertain the name of the logfile that a SAS job is writing to. The Rolling Log Options filename pattern defined in Metadata dictates the -log parameter value that is passed to the Batch Job process at runtime, which in turn sets the LOG SAS option for that process. Without this option available, and with no other way for a Macro to ascertain the exact name of the logfile from inside the SAS session and the full name of the logfile needs to be reconstructed during initialisation using some DATA Step code.

Metadata configured logfile pattern
Logfile pattern configured under logconfig.xml

This can be done by substituting the following line from the %esm_reportIn() integration macro in esmtags.sas:

logloc = getoption('log');  

to something along the lines of the following code:

data testings;
 length dd mm hh min ss $2.;
 length yyyy $4.;
  startval=datetime();
  baselogname = getoption('log');

  late = 0;
  /* Change the maximum value of the late variable 
     to suit the startup delay time of your environment   */ 
  do until (late = 10);
   testval = startval - late;
     dd=put(day(datepart(testval)),z2.);
     mm=put(month(datepart(testval)),z2.);
     yyyy=put(year(datepart(testval)),z4.);
     hh=put(hour(testval),z2.);
     min=put(minute(testval),z2.);
     ss=put(round(second(testval)),z2.);
    suffix=cats(yyyy,'.',mm,'.',dd,'_',hh,'.',min,'.',ss);
   late + 1;
   testfile = cats(baselogname,suffix,'.txt');
   if fileexist(testfile) then do;
    logloc = testfile;
   end;
  end;
 put filefound=;
run;

This code block can be found in esmtags.sas and is commented out by default. Note that the maximum value of the late variable, as set at the beginning of the do until loop, should be adjusted depending on the variability in startup time for your SAS Batch sessions. This can depend on a number of factors. For this reason it is recommended that your ESM Integration Macros are the first thing that is called by your Batch Server Autoexec file.

Per-session tracking of SAS UTILLOC size

SAS multi-threaded procedures can make use of a disk paging location referred to as UTILLOC. ESM is designed with the capability to track the size of this UTILLOC usage of each individual session, but to enable this functionality, some extra configuration is required.

Unlike the SAS WORK location, which is a directory created at session startup that is unique to each individual SAS session, SAS UTIL directories are created on-the-fly by SAS procedures as they are required. Therefore, unlike the WORK system option which holds the full path to each session's individual SASWORK directory, the UTILLOC system option points to the parent directory which can contain any number of UTIL directories belonging to a number of other sessions. There is no efficient way of deducing which UTIL directory belongs to which session, which makes periodic size measurements of a session's UTIL directory challenging.

One way of addressing this issue is to create a per-session UTILLOC subdirectory for each session before it is started using it's usermods initialisation script, and removing the directory after the session has finished executing. This results in all of a given session's UTIL directories being created under the same parent directory, making it more efficient to size and easier to clean up, and the UTILLOC option pointing to a directory dedicated to that particular session. An example way of doing this for a Workspace Server would be:

Add this to WorkspaceServer_usermods.sh to create the subdirectory and tell SAS to use it:

# set a unique subdirectory for the util loc
if [[ $ESMACTIVE -eq 1 ]]; then
  export ESMUTILLOC=/utilloc/$(hostname -s)_$$
  mkdir $ESMUTILLOC;
  export ESMUNQUTIL="-utilloc $ESMUTILLOC"
else 
  ESMUNQUTIL=
fi

set -A USERMODS_OPTIONS   # initialize empty list
# build up list by un-commenting (and adding) the lines you need (one line per token)
# then replace the <argument> with the values you want for each token on the command line
USERMODS_OPTIONS[${#USERMODS_OPTIONS[*]}]="$ESMUNQUTIL"  

Then append this to the end of WorkspaceServer.sh to clean it up once the SAS session has finished executing:

if [[ $ESMACTIVE -eq 1 ]]; then rmdir $ESMUTILLOC; fi

Warning

Unlike the other configuration changes presented in this guide, this change alters the standard SAS behaviour and is therefore not included as part of the standard ESM Configurator workflow. This change should be tested and documented. Use at your own risk.**