Performance testing can lead to proactive resolution of functional and system related issues before they impact end users and allow for identification of bottlenecks and/or underutilized resources for redeployment. To ensure a successful enterprise implementation, the following testing steps should be performed:

  • Application Functional Testing - The application functional testing is a crucial step in a successful GIS implementation. It is important that the application be complete, solid, sound and representative of defined business requirements.
  • System Tuning - Testing will expectedly reveal areas of improvement; however, some best practices should be implemented. This may include evaluation of the map documents, data location and data structure modifications.
  • Test Plan Development - A comprehensive test plan with input from domain/functional experts, documented user workflows and documented system configuration is imperative to testing success with effective and practical results.
  • Test Script Creation - Test Scripts are necessary so that tests can be re-run as required in a repeatable and controlled approach beginning with testing single user interactions and then scaling the testing to a simulated load test. Insertion of user transactions, user think time, pacing of user tasks as well as parameterizing transaction inputs are included in the scripts.
  • Testing Execution - Testing is performed using one or more load testing tools all while ensuring only target applications to be tested are running, as well as consistency in application data and the system configuration throughout the test cycles. This will produce reliable test results.
  • Analysis of Results - Analyze, compare and correlate key measures such as response times, and system behavior, evaluate unexpected results caused by test case or component failure and assess capacity thresholds and limitations.

Typically, simple web applications and services are relatively straightforward to script and performance test. However, rich client applications, e.g. ArcMap or web applications based on ADF API are more challenging to test. The following table ranks the relative level of testing for each application architecture:

Category Effort Considerations
Testing Level of Effort by Application Architecture
Rich Client Applications (ArcMap) High Requires scripting screen pixel clicks
Web Applications (JavaScript, Flex and Silverlight API) Medium-Low Stateless calls direct to the REST interfaces are easy, but emulating user movement within the application interface may be more challenging.
Web Applications (.NET and Java ADF) High

Requires managing application state and Ajax
Mobile Applications Medium  
Services Low REST, and SOAP services are the simplest to test directly

ESRI’s Testing Process

Deploy Systems

  • Validate system configurations match deployment plans
  • Functional testing
    • It is important that the application be complete, solid, sound and representative of defined business requirements and functional specifications

Tune Systems

  • Before testing ensure best practices and tuning are implemented such as
    • MXD tuning
    • Database tuning

Develop Test Plan

  • A comprehensive test plan is imperative to testing success and includes
    • Interview application specialists
    • Define workflows and transaction to be tested, including think time
    • Define target user load, e.g. concurrent users, transactions per hour, and usage patterns
    • Define weights to usage patterns 
    • Document configuration of the test environment
    • Verify success criteria
    • Define required metrics to be reported

Test Scripts

  • Develop tests script as per Test Plan specification
  • Include validation rules

Load Test Design

  • Select adequate test client machine
  • Define required metrics to be collected
  • Implement step load for virtual users
  • Implement step duration
  • Implement pacing

Test Execution

  • Initial run verifies the load test parameters yield expected behavior
  • Verify test results with additional test runs

Analysis and Report

  • The final step is to analyze, compare and correlate key measures such as response times, and system behavior, evaluate unexpected results caused by test case or component failure and assess capacity thresholds and limitations.
    • Review transaction response times
      • Identify dominant transactions by longest response times
    • Review Tier Metrics and Identify potential bottleneck Tier
      • Web servers
      • Application Servers
      • Database Servers
    • Review system metrics for each machine and identify potential hardware resource bottlenecks
      • CPU
      • RAM
      • DISK
      • Network
    • Review system and application matrix (including errors) for each tier and identify potential tier bottleneck
      • Test Client
      • Web servers
      • GIS Servers
      • Database Servers
    • Develop reports summarizing information

Addtional Support

ESRI Professional Services offers a variety of services in these technical areas. Please contact ESRI Professional Services.

Related Links

 

Filed under: