By Daniel Roberts, principal sales engineer, MarkLogic

Right now, financial market institutions are gearing up to meet the Markets in Financial Instruments Directive II (MiFID II) deadline of January 2018. MiFID II is not only another reporting requirement, it is a regulation that involves technical, political and economic requirements being put in place – requirements that run to literally thousands of pages. The long-term consequences are not entirely knowable at this time – but it is clear that non-EU firms will be impacted as well when doing business with EU entities.

The key issue is that these comprehensive regulations (with all their vagaries) and the threat of massive fines for non-compliance are creating fault lines among executives, bank risk officers and the developers tasked with complying.

As regulators bid to create more open and transparent financial markets by tackling the “under-regulated and opaque aspects of the financial system”, which will likely generate tens of terabytes of increasingly complex data, many institutions are reportedly a long way behind schedule in their efforts to meet this regulation.

This is in part because MiFID II is much larger in scope compared to the preceding regulation. It requires a different approach towards leveraging data in operational and transactional transparency to ensure investor protection and a fair and uniform functioning of the European financial marketplace.

However, addressing MiFID II is a massive data integration effort, which most financial services firms are not yet geared up for. In terms of data governance and compliance readiness, this regulation will impact transaction reporting of all financial instruments traded in Europe, post-trade transparency, and real-time data delivery. It will also require better execution with a specific need to be able to reconstruct past events and provide all types of communications related to transactions. Many financial institutions currently have more than 20,000 relational databases within their organisation, this causes huge data fragmentation and silo problems. Often what we see when these organisations approach a new data regulation like MiFID II is that their solution is to turn to yet another relational database, which further exacerbates the problem.

Patching holes won’t work. In the relational database world, current data challenges can be addressed, but thereafter organisations are required to predict the future to keep up with the changing nature of their data. This is a flawed approach; relational databases don’t cope well in environments where data structure is endlessly changing. If these organisations could effectively get a 360-degree view of their data and implement a robust data governance strategy, regulatory reporting would be a breeze, yet most organisations fail to realise this panacea.

Many of their challenges will revolve around specific requirements in MiFID on reporting and data delivery. These include:

Governance

Banks have to be able to generate the right information for the regulator, and that creates the need for a compliance library and logic that can be reused. With many consumers of information internally and externally, logic that defines fields for reporting is often different across reports and data silos. As such, data integration is therefore difficult with a relational database, as everything is focused on the syntax not the semantics – i.e. the capability to discover the facts and relationships in data, and understand the context of those facts. In a SQL database, there is no semantic mean defined for data items across the organisation and, consequently, data governance is often limited to it’s difficult to merge data for accurate, timely reporting. Rather than encourage an army of business analysts to re-process information for each data delivery, a better approach is to store and re-use methods from one central repository and associate both to the data sets and results. The methods and code can then be updated over time independently of data updates.

Transaction Reporting

For transactional reporting, MiFID II requires financial firms to send information for all eligible trades within a regulated market to a regulator as soon as practicable. This means that trades need to be reported to an authorised reporting mechanism (ARM), which is responsible for packaging up the information and providing it to the local competent authority.

Siloed controls and inflexible legacy technologies, are, in most cases, unable to handle multiple sources of data, and the number of fields for transaction reporting has significantly increased. Traditional relational databases struggle to process the data when some information doesn’t fit the required fields. That data will be rejected, resulting in inaccuracy of reportable data. The possibility to ingest the data as-is and reconcile the data in place represents significant advantages and expedites the route to compliance.

Post Trade Transparency

In a world of post trade transparency, trades are sent to the ARM in real-time through an authorised publishing authority (APA). These authorities are responsible for making public trades related to bonds, structured finance products, emission allowances, and derivatives. This is a tough requirement to satisfy.

With MiFID II, companies must demonstrate effective oversight and control over policies and procedures which govern all communications. Moreover, there is a requirement to supply regulators with communications associated with a specific trade and being able to reconstruct history of trade cycle events. Discovery or knowing what information you knew and when and how you knew it has changed over time. Now, a bitemporal view – meaning the ability to track when events occurred (valid time) in combination with when they were recorded (system time), for auditing and tracking – of data lineage should be a critical component of your regulatory reporting infrastructure.

Retail banks are very familiar with the issue of discovery, but with many messaging and data formats the ability to identify information across not just business silos but information silos (email, messages, texts, office documents etc.) is a serious challenge for investment banks. Traditional databases store unstructured data as CLOBs which, as a unit of information, is not sufficiently granular and cannot resolve complex search and discovery questions to prove how you knew the information you knew and when. Even after the search problem is solved, managing this information as part of an information life cycle management (ILM) process becomes singularly important to prevent operational costs escalating.

KPI Reporting

To safeguard against Algo trading as well as trading beyond limitations and to ensure investor protections against inducements, organisations will need to generate reports on various Key Performance Indicators (KPIs). As part of the mandate to be able to reconstruct past events and unwind any trade with communications, a second timeline will be imposed to answer the question, “What did we know when the trade was made and how can we prove it?”

Each of these challenges will impact every aspect of a bank: operations, trading and technology. The conventional wisdom will be to think “data warehouse,” “reporting solutions” and “ETL” when trying to tackle these problems, but this won’t address the underlying challenge that ETL and traditional technology provide the agility needed for organisations to change in time for January 2018.

Furthermore, organisations that view MiFID II as merely a compliance exercise will cement its failure. It is vital that financial firms establish a sound application framework that will not only model and bend to meet MiFID II’s requirement, but which will also act as a single reporting platform to help them meet any regulatory need in the future, be it MiFID II, the Dodd-Frank Act or EU GDPR.

Solving these data challenges requires a database that empowers you to integrate all of your data with minimal disruption to your business. We recommend a design approach for reporting solutions that ensures agility and flexibility. The solution should deliver a regulatory reporting platform that incorporates best practices and operational effectiveness and allow for adaptive growth in scope and scale. The design goal should not be to remedy one-off reporting requests but to build in a capability to respond to emerging requirements with relative ease and cost efficiency.

As several financial firms have already discovered, there is an easier way to bring all these data silos together. Using an operational data hub or trade store approach – built on a flexible, enterprise-grade NoSQL database with integrated Google-like search – can pay dividends for data challenges where the data and requests from regulators change over time.

ABN AMRO is using the next-gen MarkLogic database platform to bring vast amounts of unstructured and structured trade data into one central, easily manageable operational trade data store. With a consistent, transparent record of every order and trade event, ABN AMRO is able to comply with internal and external reporting requirements in a fast and flexible manner now as well as in the future.

The bottom line is that MiFID II requires new organisational processes and thinking, research and development in how to support it, and a departure from rigid technologies that make adaption costly and likely, not even feasible.