Blog Why Automated Data Workflows are a Foundational Capability for Enterprise SOCs

Why Automated Data Workflows are a Foundational Capability for Enterprise SOCs

SOAR technologies and the adoption of orchestration have fundamentally changed the way we think about cybersecurity, and we’re all better for it. Scaling through automated processes is now an option for small teams. No longer must every workflow be manual. This is helping security operations centers (SOC) make material progress in improving their organization’s cybersecurity posture while balancing a shortage of qualified and experienced analysts.

But there's another challenge starting to come into sharper focus for enterprise SOCs. Our data has gotten messier and tool sprawl is increasing. Implementing process automation and orchestration tools before your team is ready can dilute, rather than enhance, the efficacy of the SOC.

Too often I hear from security leaders who have ideal use cases for SOAR, but they can’t get their tools and datasets to talk to each other to make it all work.

As teams start taking on complex orchestration challenges, they must think through how to ingest, combine, curate and deliver data needed by various analyst tools. Automated data workflows are becoming a key enabler for enterprise SOCs to realize the full potential of SOAR. 

Bridging Gaps In Datasets 

Last year SANS surveyed  top security teams on best practices for security operation centers.  The top barriers preventing enterprise SOCs from becoming more effective included lack of automation, tool integration, and intelligence context.


Reference: Common and Best Practices for Security Operations Centers: Results of the 2019 SOC Survey, Pg. 19

In the absence of automated data workflows to connect tools and support analytical processes, analysts end up bridging data gaps with manual copy-paste operations, in effect making them glorified data entry operators, or hacking orchestration playbooks to perform complex data manipulations. Orchestration playbooks are not designed to manage your intel and using them as a data workflow engine to solve complicated data problems such as extraction, normalization, whitelisting, and prioritization is not a scalable solution.

Intelligence Management in the Era of Ecosystems

The concept of “automated data workflows” is not a new one, but its application to cybersecurity is. 

Functions like business intelligence (BI) and data science have grappled with the same challenges, especially as tool fragmentation, data scope, and data volumes increase at a faster rate than ability of existing processes to keep up with. 

Modern enterprise SOCs face a very similar challenge. SOC analysts have to organize and catalog constantly changing cyber intelligence from different sources to operationalize it in the tools they use. Alert generation and triage, event prioritization, incident investigation, and remediation are distinct tasks for a SOC analyst, each requiring its own depth of intelligence context, enrichment and structure. 

The need to account for new use cases can drive additional data enablement requirements from source to consumption. Enterprise SOCs need capabilities that cover data needs for analysts to work through a spectrum of alerts and incidents ranging from network intrusions to insider and privilege abuse, to point of sale intrusions and credential stuffing, all the way to identity breaches. Without accessible and integrated cyber intelligence, analysts are stuck manually collecting, curating, and cataloging intelligence and end up spending very little on triaging alerts, analyzing incidents, and threat hunting.

Embracing the Challenge

Automated data workflows will become a strategic imperative as SOCs move away from self-service and ad-hoc efforts to accelerate their workflows. 

Whether enterprise’s want to build their own capabilities or invest in a product to enable automated data workflows they should consider the ease/complexity of the following: 

  • Building and managing data catalogs to support a variety of use cases.
  • Addressing governance requirements about access to data and its lineage.
  • Monitoring performance of data workflows in supporting use cases and analyst actions.
  • Creating new and modifying existing data models and transformations. 
  • Making data available in a way that SOC tools can operationalize it in an automated way, without manual intervention.
  • Supporting enrichment & investigation tasks based on L1, L2, L3 analyst requirements.     

Thinking holistically about data and process workflows as part of your enterprise SOC strategy can finally start solving the challenge of operationalizing cyber intelligence and giving analysts a more complete picture of incidents in the tools of their choice. Successful SOC strategies will need to show how they can account for challenges related lack of automation, tool integration, skill shortage and ever evolving threat landscape, and automated data workflow capabilities need to become a core part of that solution.


Only the Paranoid Survive, Recast for Cybersecurity Andrew Grove's seminal business management book Only the Paranoid Survive offers a fitting title for the current state of cybersecurity and a roadmap ... Read More
The Data Dilemma in Cybersecurity Last week, the Wall Street Journal reported that the “scarcity of data needed to train models is slowing progress” toward the promise of fortifying ... Read More
The Good, Bad, and Ugly of Threat Intelligence with Patrick Coughlin Recently Co-Founder and CEO of TruSTAR, Patrick Coughlin, sat down with Ron Eddings and Chris Chocran from Hacker Valley Podcast to discuss how ... Read More
The Evolution of Intelligence in Security Operations In the last decade, threat intelligence catapulted to the forefront of security operations as companies like Mandiant and iSight Partners started to ... Read More