NeurDB

Articles
Note: This article was tagged for deletion on Wikipedia under Criteria for speedy deletion (CSD): No indication of importance (product) (A11).

| developer = NeurDB Global Development Team | released = | latest release version = | latest release date = | programming language = C, Python | genre = DBMS | website = }}

NeurDB is an AI-powered autonomous database system . It seamlessly integrates artificial intelligence into the database system, enabling intelligent AI-powered data analytics and autonomous system optimization.

AIxDB

The aspiration of integrating AI and data systems (AIxDB) was first expressed some forty years ago. With the advancement in both fields over the years and the increasing demand for "intelligent" data systems offering advanced analytics, the fusion of both technologies has become more imminent. In 2016, the mutual benefits that deep learning and data systems can offer each other were further elaborated.

Related data systems have explored extending relational query processing with machine learning inference. In these systems, machine learning pipelines can be executed within database queries, providing more advanced data analytics. Further, machine learning methods can also be used to improve internal system components such as query optimization., indexing, and concurrency control

The design of NeurDB follows the AIxDB paradigm, which aims to support AI-powered analytics and autonomous database optimization within a unified system.

Development

NeurDB was introduced by researchers from the National University of Singapore and Zhejiang University as part of research exploring AIxDB.

The early development of NeurDB began with the system positioning with key features and key techniques. They first positioned how to deeply integrate machine learning ecosystem into data system kernal and how learned system components could adapt to data and workload drift.

Following the initial positioning, the project evolved into a proof-of-concept prototype that explored the execution of AI workloads inside relational databases. This prototype proposed an in-database AI engine that can directly execute machine learning inference and training in database execution engines, as well as enabling database components to adapt to data and workload drift.

Subsequent development expanded the system to incorporate AI techniques into several core database components, including query optimization, concurrency control, indexing, and model management. These efforts aimed to build a unified AI-native database architecture capable of supporting both transactional workloads and AI-powered analytics within the same system.

NeurDB is open source, and its development is still ongoing, with new components and more advanced analytic features.

Release milestones

NeurDB has the following releases:

Key features

ACID-compliant transactions

NeurDB maintains ACID guarantees. By extending the architecture of the relational database PostgreSQL, the system preserves transaction correctness.

AI-powered analytics

NeurDB supports AI-powered data analytics directly with an extended SQL interface.

A core statement in NeurDB is PREDICT:

PREDICT { CLASS | VALUE } OF <column>
FROM { <table> | (<subquery>) }
TRAIN ON { * | <feature_columns> };

For example:

PREDICT VALUE OF customer_class
FROM (SELECT * FROM customers)
TRAIN ON age, income;

When executed, NeurDB can automatically finish such an AI query. In particular, it trains the model if necessary and performs inference on the specified data. The prediction results are returned as tables.

Autonomous optimization

NeurDB incorporates learned database components that enable autonomous system optimization. Components such as query optimizers, transaction processing, and indexes are enhanced with machine learning to adapt to data and workload drift.

Secure AIxDB

NeurDB follows the AIxDB architecture paradigm, which integrates AI workloads and database systems within a unified platform. The system therefore supports secure and privacy-aware execution for both AI and data within the database environment.

Architecture

NeurDB Architecture

NeurDB consists of an AI analytics layer, a database engine, adaptive data access components, and a storage layer, together forming an AI-powered database architecture.

AI analytics layer

The AI analytics layer, referred to as NeurIDA, manages AI-powered analytics tasks within the database system. It provides mechanisms for advanced data analytics, including model selection, model slicing, and model construction.

Database engine

The database engine enables AI-powered query processing and system optimization.

Adaptive data access components

To improve data access efficiency under dynamic workloads, NeurDB includes adaptive data access components.

Storage layer

The storage layer manages both heterogeneous data and machine learning models.

The NeurStore subsystem provides efficient storage and lifecycle management for deep learning models within the database. It supports tensor-level deduplication and compression-aware model loading to reduce storage overhead and improve model loading performance.

The storage layer also supports heterogeneous data storage formats, including key-value storage and heap storage.

System utilities and agents

NeurDB includes additional system utilities and agents designed to coordinate and evaluate AI-powered database functionality.

References