Imply Data

In 2011, the Druid project was started at Metamarkets, an online advertising company now part of Snap, to power an analytics product. Druid was open sourced in October 2012 under the GPL license.[2][3] Over time, notable organizations including Netflix[4] and Yahoo[5] adopted the project into their technology stacks. The increased adoption led the team to change the license of the project to Apache.[6] With the growing popularity of the open source project, the creators of the project decided to form a company to advance the uses of Druid.

Imply was founded in 2015 by three of the co-creators of Apache Druid, Fangjin Yang, Gian Merlino and Vadim Ogievetsky (who is also a co-creator of D3.js). The three had worked together to create Druid to support the need for real-time exploratory analytics on large data sets.[7]

In October 2015, Imply announced that it had raised $2 million from Khosla Ventures.[8] and launched its first product, combining Apache Druid and additional open-source components, including a user interface and the PlyQL SQL-like query language, plus enterprise support.[9]

In December 2019, Imply announced that it had raised a Series B round of an additional $30 million at a valuation of $350 million.[10] The funding round was led by Andreessen Horowitz with participation from Khosla Ventures and Geodesic Ventures.[11]

Imply Pivot, a prebuilt visualization application for intuitive data exploration was launched in 2020.[12]

A Series C round of $70 million, valuing the company at $700 million, was announced in June, 2021, led by Bessemer Venture Partners.[13]

In November, 2021, the fourth co-creator of Druid, Eric Tschetter, joined Imply as Field Chief Technology Officer.[14]

Also in November, 2021, Imply announced Project Shapeshift, designed to develop a hardware-abstracting, auto-scaling control plane and SaaS service for Apache Druid, extend the Druid SQL API from querying to ingestion, processing & transformation and build a serverless and elastic consumption experience.[15]

Druid is an open-source database, distributed under an Apache license since 2013.  Imply provides support, management, monitoring, the production-ready containers to simplify deployment and operations of Druid.[16]

Imply also provides services to deploy and manage Druid in the cloud, using Amazon Web Services.

Imply Pivot is a visualization engine for Druid.

Imply is a commercial distribution of open-source Druid, and shares the same common use cases: a database where real-time ingestion, fast query performance, and high uptime is important. [17]

Airbnb uses Imply to collect, organize, and process a deluge of data (all in privacy-safe ways), and empower various organizations across Airbnb to derive necessary analytics and make data-informed decisions from it. Ingestion comes from both Hadoop sources of historical data and Kafka sources of streaming data, while visualization is provided by Apache Superset.

Dream11’s Inhouse Analytics using Imply to understand 3 billion daily events totaling about 4.5TB per day, analyzing the full set of data instead of depending upon data sampling while maintaining data security and providing accelerated reporting.[18]

Walmart uses Imply for low-latency ingestion and extremely fast integration from Kafka and Storm to make it easy for the people across the organization to access event data from over 11,000 stores and online sites, analyze it, and make decisions in as short of a time as possible.

GameAnalytics ingests real-time data on over 15 billion gaming events daily to provide user behavior analytics for video game developers, with data from game SDKs streamed with Amazon Kinesis to Imply Cloud, providing reliability, low query latency and flexible querying at a low infrastructure cost.[19]

Imply can also be used for data preparation for data science at scale.[20]

Reddit uses Imply to allow advertisers to query across both current and historical data performing specific aggregates and breakdowns with hundreds of billions of raw events. New data is ingested directly from Kafka, providing results in near-real time.[21]

In May, 2019, José Correia, Carlos Costa, and Maribel Yasmina Santos published Challenging SQL-on-Hadoop Performance with Apache Druid [22]at the 22nd International Conference on Business Information Systems.[23] They compared performance of Hive, Presto, and Druid using a denormalized Star Schema Benchmark based on the TPC-H standard. Druid was tested using both a “Druid Best” configuration using tables with hashed partitions and a “Druid Suboptimal” configuration which does not use hashed partitions.  

Tests were conducted by running the 13 TPC-H queries using TPC-H Scale Factor 30 (a 30GB database), Scale Factor 100 (a 100GB database), and Scale Factor 300 (a 300GB database).

Druid performance was measured as at least 98% faster than Hive and at least 90% faster than Presto in each scenario, even when using the Druid Suboptimized configuration.

In November, 2021, Imply published the results of a benchmark using the same Star Schema Benchmark, running using Druid on an AWS c5.9xlarge instance at Scale Factor 100 (a 100GB database). The 13 queries executed in a total of 0.747s.[24]

Notable customers of Imply include

• GameAnalytics[25]
• British Telecom[26]
• Dream11[27]
• TrafficGuard[28]
• Outbrain[29]
• Twitch[30]
• Twitter[31]
• Reddit[32]
• Innowatts[33]
• Adikteev[34]
• Sift[35]
• WalkMe[36]
• Airbnb[37]
• Walmart[38]
• Charter Communications[39]
• Zscaler[40]
• Ibotta[41]
• DBS Bank[42]
• Blis[43]
• Expedia[44]
• Lyft[45]
• NTT Global IP Network[46][47]
• TripleLift[48]
• TrueCar[49]
• Reddit[21]

As Imply Enterprise uses Apache Druid as its database engine, it shares the limitations of Druid.

SQL for queries only: Druid uses its own native query language. It also supports SQL queries, with a parser and planner based on Apache Calcite. Only SELECT statements are supported, not other SQL commands such as INSERT.[50]

Limited support for SQL Joins: Until the release of Imply 3.3 supporting Apache Druid 018, Imply had no support for table joins and all data had to be denormalized before ingestion. Current releases support joins, but only joining small tables to one another or small tables to a single large table (a star schema). Only left joins and inner joins are supported, and there is a cost in query latency for any joins in a query.[51]

Ingestion Complexity: loading data into Druid can be complex, requiring a JSON specification document to define ingestion from both streaming sources (Apache Kafka, Amazon Kinesis, or Tranquility) and batch sources (Amazon S3, Azure Blob, Google Cloud Storage, Hadoop HDFS, and others).[52]

No SaaS: while Imply offers Imply Cloud with preconfigured systems managed on Amazon Web Services, it does not offer a cloud-native Software-as-a-Service option that removes tuning parameters, abstracts hardware boundaries for auto-scaling, integrates natively with other cloud services, and enables usage-based pricing. Future support for Imply SaaS was announced as part of the Project Shapeshift announcement at the Druid Summit in November, 2021.[53]

• Official website
• Products
• Apache Druid website