Towards SOC Compliance

Continuous business value creation while handling customers’ data in a secure way is a must for SaaS (Software as a Service) providers to survive in today’s rapidly changing, highly coupled and deadly competitive business environment. Even though the organizations follow their own techniques and processes, customers expect some kind of mutually recognized guarantee to ensure that their data is handled in a secure manner. Achieving Service Organization Control(SOC) compliance is one such guarantee. Hence, SaaS providers may at least achieve SOC 2 Compliance to ensure their customers that their services meet data security requirements.

Being a Technology Solution company who wants to get emotionally touched with their clients (SaaS providers), it is important to implement some reports and adhere to some procedures to make sure our clients achieve SOC2 Compliance certificate. However, the reports that are needed (Eg: User Snapshot Report, Change Client Report, Failed Login Report … ) and processes to follow (Eg: Selected Sensitive Fields Encryption, Employee Deletion & Obfuscation with GDPR, scrubbing data, decommissioning canceled clients, maintain audit logs) will be decided by the service providers based on their specific business practices to comply with selected trust service principles out of five trust service principles.

Pic 1 : FiveTrust Service Principles

(Note: SOC related trust service principles are security, availability, processing integrity, confidentiality and privacy. For SOC2 compliance it is not a must to have all of them in place)

Readiness Testing

Even Though SOC2 audit is done by an independent CPA (Certified Public Accountant) or accountancy organization, we (QA engineers) also can do a basic readiness testing by verifying the newly created reports/scripts/logs, sensitive data encryption etc… These small actions will provide significant confidence to SaaS providers to move forward with SOC2 audit as it reduces the risk of exceptions and failures. Furthermore, QA team needs to make sure the newly added changes do not affect the functionality and the performance of the application.

Validating Sensitive Fields Encryption

Encrypting all the identified sensitive fields is an easy way to achieve two trusted principles, privacy and confidentiality. Data can be encrypted as required after proper identification of all the tables (in both systemDB and clientDB) which include the identified sensitive fields. Then those tables/fields can be validated by querying as expected to check whether the ciphertext is shown in the required fields, instead of actual data. If you still want to check whether the correct data is stored as ciphertext, that can be easily done using SSMS 2017. A certificate will be created on the database server when identified fields are encrypted using ‘Always Encrypted’. Then by exporting that certificate, and import it to your machine can help you to view both exact data and ciphertext based on the mode selected.

Validating Intrusion Detection and Prevention

Intrusion detection and prevention can be done via network and application firewalls, custom rules and implementing two-way authentication to achieve security. Implementing two-factor authentication requires users to provide two ways of verification when logging into an account, such as a password and one-time passcode (OTP) sent to a mobile device. It strengthens intrusion prevention by adding an extra layer of protection to the application’s sensitive data. Verification can be done using an application that supports time-based one-time password token generation (TOTP token Generators). Permission structure, UI validations, reset password flow, enabling and disabling 2FA, are few areas that need to be taken into consideration when testing 2FA.

Maintaining a Failed Login Report will also help to keep track of all the failed login attempts. This information can be used to investigate and make quick and informed decisions about detected intrusions.

As QA engineers we need to make sure that failed login report includes accurate and relevant information related to all failed login attempts for all types of user accounts regardless of active/inactive status. Related data can be stored in the Eventlog table. Related xml can be viewed by querying as needed.

Pic 2 : Sample xml related to failed logon report

Testing Audit Logs

To achieve SOC compliance, monitoring authorized and unauthorized system configuration changes, and user access levels changes is essential. Adding/ Editing permissions of user access levels, changing already defined permission structures are few activities that need attention. QA team can validate the related xml and its content by performing such activities.

Pic 3 : Sample xml related to changing report permissions

Validating Decommissioning Cancelled Client Process

Decommission inactive/canceled clients from the application and remove all historical data is essential to satisfy security requirements around data retention policies. The development team can come up with a script to remove the data from the system database. This script can be given to SaaS providers so that they can execute it when they need to decommission the canceled or inactivated clients.

Following steps can be used to validate the Decommissioning Cancelled Client Process,

  1. Inactive / cancel the client (via application /using their license)
  2. Execute the script to remove the historical data
  3. Verify no record is there in the systemDB after decommissioning (You can basically check deleted clientID do not exist in systemDB )

Apart from just validating reports/xmls, as proactive QA engineers we need to follow up on modifying the scrubbing scripts, sensitive data encryption, event logs and audit logs with their xml with the schema modification. These actions will help to increase our value significantly.

Moreover, if we want our clients to achieve SOC 2 compliance, we need to adhere to the agreed process, because in SOC audit even the process is validated. Below mentioned few examples for such agreed rules/process in terms of quality assurance.

  • All the production candidates should be QA verified to do a production release.
  • All the changes should be approved by an authorized party.
  • QA sign off should be given as a document not via email and that document should be attached to the production release ticket.

Performing end to end SOC compliance testing cannot be done by QA engineers, especially the offshore QA team. However, they can test the newly implemented two-factor authentication, sensitive fields encryption, audit logs, scrubbing scripts, decommissioning canceled clients scripts and few other reports related to intrusion detection, disaster recovery and security incident handling.

These small actions will provide great value addition for the customers by increasing significance confidence to proceed with SOC2 audit. So why not proceed with these few step

These small actions will provide great value addition for the customers by increasing significance confidence to proceed with SOC2 audit. So why not proceed with these few steps.


Nuwani Navodya

Associate Lead QA Engineer

Take Maximum Out of Quality Metrics

Have you ever felt calculating and presenting quality metrics as a useless, time-consuming cumbersome task? If your answer is yes, then it seems you are not using the correct quality metrics to measure the quality of your application/release build or you are not aware of how quality metrics can be utilized to improve the quality as well as taking decisions.

This article will provide you some insights on the importance of carefully selected and well-presented quality matrices. Reading this article will motivate you to use quality metrics smartly.

Why ‘Quality Metrics’ Matter?

Quantitative approach of measuring the quality with proper guidelines is essential to compare the quality of two builds or two projects as quality is subjective and differs from person to person. Isolating test execution cannot measure and improve the quality of a software. Carefully selected quality metrics need to be used to evaluate the quality aspects by identifying the initial level of quality, current quality status, deviations and quality achievements.

Apart from that, nowadays software companies tend to push their software to the market as fast as they could to gain competitive advantages. This behavior increases the risk of releasing a highly defective release to production environment. Quality metrics can provide a significant help to control the above, as quality cannot be compromised at any cost.

Following are some of the usages of quality metrics.

  • Influence the decision-making
  • Set quality goals
  • Evaluate the testing effort
  • Report the progress of the project
  • Communicate an issue
  • Foresee the risks using the deviations from the quality goals and take preventive actions

How to choose the right quality metrics?

Selection of right quality metrics is crucial to get the maximum out of the quality metrics. Context, purpose/usage, traceability and audience are four key factors when selecting a metric. Apart from that having a common interpretation, actionability, accessibility and transparency are few other qualities of a good metric.

Link :

Selection of too vague metrics does not provide any actionable insight whereas too specific metrics are not comprehensive enough to represent the actual difference made to overall product quality. To derive the most benefit from metrics, it is important to keep them simple and relevant.

How quality metrics can be used to improve the quality?

Evaluating the quality of the software under development is the primary objective of using quality metrics. Quality of the system under test can be improved by identifying issues, analyze progress/trends and take corrective actions efficiently and effectively before it is too late.

Apart from that evaluating the current testing process and the testing strategy is one of the key benefits of using quality metrics. Metrics related to testing effort, test effectiveness, test coverage, test execution rates, defect finding rate etc. can provide valuable information about test progress and effectiveness of current testing methods. This information can help the quality assurance team to make relevant changes to the current testing process.

Furthermore, quality metrics can also be used to challenge the development team by setting high quality targets (Eg: Defect Severity Index should be less than 1.5 to pass the build) that can ultimately result in high process and product quality.

How Quality Metrics can be utilized to
take managerial level decisions?

Usage of proper quality metrics increases the confidence of taking managerial level decisions.
Given below are some decisions that can be taken by observing the quality metrics.

  1. Development process improvements related decisions as quality metrics provide complete visibility of the effectiveness of software development efforts.
  2. QA process and strategy related decisions by observing how QA efforts add value to deliver high-quality software.
  3. Resource utilization related decisions
  4. Technology change related decisions by observing defect related metrics such as defect age, defect removal efficiency etc.

Accurate and timely details presented in an effective manner is vital to take management decisions. Having a central dashboard with up to date quality metrics provides great value to the decision-making process as quality metrics together can provide valuable insights than using it individually.

Build Quality Index is kind of a dashboard which presented quality-related metrics such as Defect Density, Defect Severity Index, Defect Status Distribution (DSD), Test Pass Ratio (TPR) related trend charts are presented on a dashboard concerning each build etc.

In conclusion, software quality metrics provide immense value to increase the quality of application /release build and overall decision making process. Hence, let’s pay more attention on calculating and presenting software quality metrics to take the maximum benefits out of them.

Nuwani Navodya

Associate Lead QA Engineer