Streaming Data Security and Governance

By Admin
31st, Mar 2020

Part One

Financial businesses are increasingly operating in a real-time environment. They now need to reconcile with large data produced continuously from various sources such as Mobile Apps, Web Clickstreams, Application Logs, IoT Sensors, and so on. The value of streaming data used in real-time decision-making diminishes or perishes, very quickly. With the briefest of the passage of time such data quickly becomes historical data that is generally suitable to be processed by traditional batch jobs for business intelligence type applications. For governance, risk and security function to be successful in “real-time” enterprises, we increasingly find that applying streaming big data and machine learning technology-based solutions are required – to effectively manage high-speed data at scale.

In the security domain, as the severity of attacks continues to increase the damages and costs are spiraling out of control. The damage to the brand, and the years lost in rebuilding the image and reputation, and recovering from the financial losses present enormous challenges to many businesses. Additionally, given the evolving nature of security threats and the massive inventory of credentials up for sale on the black market, most attacks involve the illegitimate use of a legitimate account. The eroding organizational boundaries and the lack of governance mechanisms for streaming data are leading to a situation where there is increasing distrust (the fourth V – veracity) in using such data in decision-making (leading to further losses).

Architectures and development processes are also evolving in financial institutes to include a mix of on-premise and cloud infrastructure, internal and external data sources, big data storage and application frameworks, and legacy and new-age applications such as streaming analytics, machine learning applications, neural networks, big data visualization tools, and agile process-driven environments. In this scenario, implementing security and governance measures becomes a much more challenging and complex task.

Though, integration of varied data sources, systems, big data technologies, advanced analytics, and ML / DL driven applications have been the mantra in the business applications space for a while now, there’s been little progress towards integrating disparate point solutions used in security operations or using integrated big data security analytics security and risk management. There are some big data-based security and governance solutions however very few that can be used for meeting real-time security, risk, and governance of high-speed streaming data. (Continued)