IBM Patent May Help Clean Data Pipelines

Not all data is created equal. 

IBM may be looking for ways to sift out the best: The company filed a patent application for “modifying data pipeline(s) and dataset quality using data observability” that would monitor and improve the quality of data pipelines in real time. 

The system continuously collects observability data from all data pipelines within a company’s architecture, checking criteria such as data quality and lineage, error history within the data pipelines, and resource consumption. Each of the datasets created from those pipelines is classified based on reliability of the data and usage, or how critical it is to operations. 

Based on the classifications and observability metrics, the system then generates recommendations for how to improve the data pipelines or the quality of the data itself. This kind of real-time monitoring allows for preventative maintenance of data pipelines and creates more trustworthy datasets for a variety of use cases. 

“Well-organized data pipelines provide the foundation for a range of data projects; this can include exploratory data analyses, data visualizations, and machine-learning tasks,” IBM said in the filing.

Data sits at the core of all AI models: The better the data, the better the model will work. And as enterprises navigate their AI strategies, whether they’re developing proprietary models or fine-tuning public ones, getting their hands on large amounts of high-quality data is imperative. Tools like IBM’s could streamline that process. 

Patents like these add to a number from IBM and other tech firms that seek to offer peripheral technologies to support AI adoption and development. While IBM doesn’t sit among major model providers, developing and snagging IP for tools that help carry this tech transformation could prove to be lucrative as adoption marches on.