Healthcare

Information retrieval is difficult because it can come in multiple different forms. It can come in the form of voice, text, and images. It can also come from implicit searches. 

Document retrieval is matching a user query against a set of free-text records that can include unstructured text, such as newspaper articles, real estate records, manuals, financial documents, and records held by banks and other financial institutions or healthcare organizations. The point of information retrieval is to provide the searcher with the correct document. Document retrieval is a crucial step because it involves retrieving information that you will need to apply to the present or future. Querying usually involves returning the same type of information that was searched, but it can sometimes be the opposite. If the query is not clear, the information retrieval system may employ user history, physical location,  and other context clues to return the proper results. Retrieving information can include ranking existing pieces of content, such as documents or short-text answers, or composing new responses incorporating the retrieved information.

Challenges

Information retrieval is vital in a world where documents are being digitized at a rapid pace. Malfunctioning retrieval systems can lead to a massive loss in efficiency. Another major challenge is summarizing large documents and web pages.

Searchers can also choose to search using metadata, such as document titles, creation date, keywords, even the creator. If a user-stated query needs more information on the context of the content in the documents or details of the background of the content, metadata-based search and information retrieval would not be a viable solution. For example, if a user-stated query includes historical information and needs knowledge from a chart or graph seen in a financial proposal, a metadata-based search would be insufficient to provide the required information. Also, it is a huge task to visualize an image embedded in a document into an easily retrievable natural language search query or term by using merely metadata as it would take years of effort in indexing images meaningfully.

Solution Approach

xpressso.ai platform provides out-of-the-box development frameworks.  The project was started with the relevant environments, which were then created automatically. Development images configured based on pre-defined templates were installed on-premises or in a development VM within the xpresso.ai infrastructure. This enabled authentication using LDAP, seamless project setup using Bitbucket, Jenkins, and Docker (ensuring build and deployment without software compatibility issues).  

The platform made available by xpresso.ai leverages the latest ML and DL tools while preparing models. It includes Pachyderm-based data versioning, Kubernetes, Kubeflow, and Spark-based ML and DL. It also includes an Istio-based service mesh-enabled microservice architecture, and ELK-based monitoring capability, contributing to a reduction in latency time.

xpresso.ai’s MLOps platform allows establishing high-end Alluxio and Presto-based efficient data connectivity and collecting data from diverse sources. A major part of the data transformation journey while creating models involved setting up the required infrastructure and collecting raw, continuous, unformatted, unparsed data from many sources. Crawlers were used to extract data from these sources like internal documents and external websites such as Bloomberg, BlackRock, etc. 

The details collected were added as exploratory variables by using xpresso.ai libraries and analyzed. By using xpresso.ai data versioning and xpresso.ai connectivity libraries, data versions were easily controlled and stored into xpresso Data Model (XDM)-enabled data store. This enabled easy retrieval and storage of datasets/ files into internal XDM. 

The attributes obtained were used for categorization (employing Pachyderm-based data versioning) and then performing univariate, bi-variate, and Bag of Words analysis — for both structured and unstructured datasets through xpresso Exploratory Data Analysis (Data and Statistical Analysis).  Different datasets and their different versions were easily controlled and stored into an xpresso Data Model (XDM)-enabled data store that enabled easy retrieval and storage of datasets/ files into internal XDM. This was achieved using two excellent features of xpresso.ai: 

1. Data Connectivity Marketplace libraries
2. Data Versioning

The xpresso Data Pipeline Management (Rapid Model Training and Experimentation) uses Kubeflow-enabled pipelines. Thus, multiple experiments using different models and datasets could be created, tested, paused, and restarted to gain better insight. 

A type of Deep Neural Networks (DNN) was used for Convolutional Deep Structures Semantic Models (CDSSM), and a combination of Convolutional and Recurrent Neural Networks (CNN + RNN) with Vector Space Models (VSMs) being the AI footprint which served as the basis of the DNN. 

It led to an effective document retrieval system being created from articles, blogs, millions of internal as well as external documents (PDFs, business and financial news, reports, etc.) by querying the corpus and retrieving relevant paragraphs and sections from reference materials. It enabled over a 90% productivity enhancement for clients looking to search relevant business content from an existing (and growing) database.