Kdb+ and the Internet of Things/Big Data

Kdb+ is a column-based relational database with extensive in-memory capabilities, developed and marketed by Kx Systems. Like all such products, it is especially powerful when
it comes to supporting queries and analytics. However, unlike other products in this domain, kdb+ is particularly good (both in terms of performance and functionality) at processing, manipulating and analysing data (especially numeric data) in real-time, alongside the analysis of historical data. Moreover, it has extensive capabilities for supporting timeseries data. For these reasons Kx Systems has historically targeted the financial industry for trading analytics and black box trading based on real-time and historic data, as well as realtime risk assessment; applications which are particularly demanding in their performance requirements. The company has had significant success in this market with over a hundred major financial institutions and hedge funds deploying its technology. In this paper, however, we want to explore the use of kdb+ for big data and Internet of Things based use cases.

Kdb+ is a column-based, hybrid in-memory database with stream processing capabilities, primarily designed for analytic workloads. In so far as in-memory capability is concerned, we refer to it as “hybrid” because it uses in-memory processing as much as it can but recognises that in some cases it may be impracticable to load all relevant (typically historic) data into memory and that you therefore need to employ techniques that will not only leverage memory-based processing but optimise performance when not all the data can fit into memory. This is similar to the approach taken by IBM with its DB2 BLU Acceleration, as an example. The product’s stream processing capabilities (which means that you can analyse very large quantities of information in-flight, in real-time) arise from the fact that kdb+ is tightly integrated with the product’s development language, q. This is a vector (array) processing language that is much more efficient than SQL and can be used to develop analytic applications as well as for query purposes.