Blog Why Automated Data Workflows are a Foundational Capability for Enterprise SOCs

Why Automated Data Workflows are a Foundational Capability for Enterprise SOCs

SOAR technologies and the adoption of orchestration have fundamentally changed the way we think about cybersecurity, and we’re all better for it. Scaling through automated processes is now an option for small teams. No longer must every workflow be manual. This is helping security operations centers (SOC) make material progress in improving their organization’s cybersecurity posture while balancing a shortage of qualified and experienced analysts.

But there's another challenge starting to come into sharper focus for enterprise SOCs. Our data has gotten messier and tool sprawl is increasing. Implementing process automation and orchestration tools before your team is ready can dilute, rather than enhance, the efficacy of the SOC.

Too often I hear from security leaders who have ideal use cases for SOAR, but they can’t get their tools and datasets to talk to each other to make it all work.

As teams start taking on complex orchestration challenges, they must think through how to ingest, combine, curate and deliver data needed by various analyst tools. Automated data workflows are becoming a key enabler for enterprise SOCs to realize the full potential of SOAR. 

Bridging Gaps In Datasets 

Last year SANS surveyed  top security teams on best practices for security operation centers.  The top barriers preventing enterprise SOCs from becoming more effective included lack of automation, tool integration, and intelligence context.


Reference: Common and Best Practices for Security Operations Centers: Results of the 2019 SOC Survey, Pg. 19

In the absence of automated data workflows to connect tools and support analytical processes, analysts end up bridging data gaps with manual copy-paste operations, in effect making them glorified data entry operators, or hacking orchestration playbooks to perform complex data manipulations. Orchestration playbooks are not designed to manage your intel and using them as a data workflow engine to solve complicated data problems such as extraction, normalization, whitelisting, and prioritization is not a scalable solution.

Intelligence Management in the Era of Ecosystems

The concept of “automated data workflows” is not a new one, but its application to cybersecurity is. 

Functions like business intelligence (BI) and data science have grappled with the same challenges, especially as tool fragmentation, data scope, and data volumes increase at a faster rate than ability of existing processes to keep up with. 

Modern enterprise SOCs face a very similar challenge. SOC analysts have to organize and catalog constantly changing cyber intelligence from different sources to operationalize it in the tools they use. Alert generation and triage, event prioritization, incident investigation, and remediation are distinct tasks for a SOC analyst, each requiring its own depth of intelligence context, enrichment and structure. 

The need to account for new use cases can drive additional data enablement requirements from source to consumption. Enterprise SOCs need capabilities that cover data needs for analysts to work through a spectrum of alerts and incidents ranging from network intrusions to insider and privilege abuse, to point of sale intrusions and credential stuffing, all the way to identity breaches. Without accessible and integrated cyber intelligence, analysts are stuck manually collecting, curating, and cataloging intelligence and end up spending very little on triaging alerts, analyzing incidents, and threat hunting.

Embracing the Challenge

Automated data workflows will become a strategic imperative as SOCs move away from self-service and ad-hoc efforts to accelerate their workflows. 

Whether enterprise’s want to build their own capabilities or invest in a product to enable automated data workflows they should consider the ease/complexity of the following: 

  • Building and managing data catalogs to support a variety of use cases.
  • Addressing governance requirements about access to data and its lineage.
  • Monitoring performance of data workflows in supporting use cases and analyst actions.
  • Creating new and modifying existing data models and transformations. 
  • Making data available in a way that SOC tools can operationalize it in an automated way, without manual intervention.
  • Supporting enrichment & investigation tasks based on L1, L2, L3 analyst requirements.     

Thinking holistically about data and process workflows as part of your enterprise SOC strategy can finally start solving the challenge of operationalizing cyber intelligence and giving analysts a more complete picture of incidents in the tools of their choice. Successful SOC strategies will need to show how they can account for challenges related lack of automation, tool integration, skill shortage and ever evolving threat landscape, and automated data workflow capabilities need to become a core part of that solution.


The Evolution of Intelligence in Security Operations In the last decade, threat intelligence catapulted to the forefront of security operations as companies like Mandiant and iSight Partners started to ... Read More
Toward MTTD & MTTR as North Star Metrics Data-centric security leaders from across industries have embraced Mean-time-to-Detection (MTTD) and Mean-time-to-Resolution (MTTR) as key metrics ... Read More
How TruSTAR Uses MTTD and MTTR as North Star Metrics The north star metrics for Data-Centric Security Automation are the minimization of MTTD (Mean Time to Detection) and MTTR (Mean Time to Resolution). ... Read More
Customer Spotlight Webinar: PVH Corp. TruSTAR has a new webinar series, Customer Spotlight, where we invite security leaders and operators from across the TruSTAR community to share ... Read More