Self-host Druid
druid-38.0.0Real-time OLAP database for sub-second analytics at scale
- Stars
- 14 097
- RAM
- 6 GiB (Docker quickstart cluster: at least 6 GiB available for Docker)
- Ports
- 8888 (Router process serving the web console), 8081 (Coordinator service), 8082 (Broker service (queries)), 8083 (Historical service (querying segments)), 8091 (MiddleManager service (ingestion tasks))
- Requires
- Apache ZooKeeper
apache/druidWhat is Druid?
Apache Druid is an open-source analytics database built for real-time OLAP workloads, aiming to shorten the gap between an event happening and an insight being acted on. It takes in streaming feeds through native connectors for Apache Kafka and Amazon Kinesis, and batch data from stores such as HDFS or S3, storing columnar, time-sorted, dictionary- and bitmap-encoded segments that answer aggregate queries in milliseconds even over extremely high-cardinality datasets with enormous row counts. Users query it with SQL via HTTP or JDBC, or through the built-in web console, which bundles a query workbench, a point-and-click ingestion wizard and cluster-oversight views powered by SQL system tables. A deployment is a set of cooperating services (coordinator, broker, historical, middleManager, router) plus ZooKeeper for coordination, a metadata database such as PostgreSQL and a deep-storage layer; the docs describe tiered storage with prioritized quality guarantees, loosely coupled scale-out, automatic replication, continuous backup and automated recovery. Optional HTTP basic, LDAP or Kerberos authentication, role-based authorization and TLS exist but are off by default and must be configured for production. The quickstart runs as a Docker Compose stack of eight containers, with 6 GiB or more expected to be allocated to Docker; production setups go on commodity *NIX hardware or in clouds, and Kubernetes installs use a separately maintained operator. Druid presents itself as a FOSS substitute for conventional warehouses, not as a text-log search system or a pure metrics store.
Druid alternatives
databendSelf-hosted, open-source data warehouse in Rust for large-scale analytics plus vector and full-text search, with Python UDF sandboxes and git-like data branching.Comparedatabend with Druid
automqSelf-hosted, open-source Apache Kafka fork with stateless brokers that store logs on S3 or MinIO and rebalance partitions in seconds.Compareautomq with Druid
cloudberryOne advanced and mature open-source MPP (Massively Parallel Processing) database. Open source alternative to Greenplum Database.Comparecloudberry with Druid
Apache IoTDBApache IoTDB is a self-hosted, open-source time-series database for industrial IoT with high-throughput reads/writes, ultra-high compression, and SQL-like queries.CompareApache IoTDB with Druid
LibreDB StudioSelf-hosted, open-source SQL IDE covering 41 database engines in one browser tab, with OIDC SSO, RBAC, query auditing and an AI query agent — MIT licensed.CompareLibreDB Studio with Druid
questdbSelf-hosted, open-source time-series database with SAMPLE BY and ASOF join SQL, QuestDB Wire Protocol ingestion, PGWire support and a web console.Comparequestdb with Druid
timeplusSelf-hosted, open-source streaming SQL engine for real-time analytics and AI — a lightweight alternative to ksqlDB and Flink powered by ClickHouse.Comparetimeplus with Druid
clickhouseClickHouse is an open-source, self-hosted column-oriented DBMS for real-time OLAP analytics using SQL.Compareclickhouse with Druid
All self-hosted alternatives to Amazon Redshift → All self-hosted alternatives to Google BigQuery → All self-hosted alternatives to Snowflake →
Should you self-host Druid?
Druid scores 75/100 on selfhost.directory (how scoring works) and ranks #1 of 2 in Data Warehousing. It has 14 097 GitHub stars, +35 in the last 30 days.
It shipped 5 stable releases in the past year. The latest, druid-38.0.0, came out on 24 September 2026. New versions land roughly every 2 months.
Setup is advanced: plan for several services and some ongoing maintenance. An official Docker image is available. It is Apache-2.0-licensed and written mostly in Java.
Its closest alternative on selfhost.directory is openobserve (79/100, 23 066 stars). It scores higher overall, so compare the two before you pick. See Druid vs openobserve side by side.
Features
- Batch Ingestion
- Built-in web console
- Cluster management views
- Columnar compressed storage
- High availability and self-healing
- Horizontal Scaling
- Joins
- REST API
- Role-based authorization
- Schema auto-discovery
- SQL Support
- Streaming Ingestion
- Tiering and quality of service
- TLS EncryptionDisabled by default; must be configured for production.
Druid README
How to install Druid
Install Druid with Docker Compose
DRUID_XMX, DRUID_XMS, DRUID_MAXDIRECTMEMORYSIZE, DRUID_SINGLE_NODE_CONF, DRUID_LOG4J, DRUID_LOG_LEVEL, JAVA_OPTSCreate a directory to hold the Druid Docker files. Download the example docker-compose.yml and the example environment file into it, then:
cd into the directory that contains the configuration files.
Run docker compose up to launch the cluster with a shell attached, or docker compose up -d to run the cluster in the background. Once started, navigate to the web console at http://localhost:8888.
Example stack launches eight containers: Zookeeper, PostgreSQL and six Druid containers.
How to configure Druid
druid.enableTlsPortTurn on TLS globally in common.runtime.properties.values: true/false
DRUID_XMX=1 GBSet Java Xmx, the maximum heap size.DRUID_XMS=1 GBSet Java Xms, the initial heap size.DRUID_MAXDIRECTMEMORYSIZE=6 GiBSet Java max direct memory size.DRUID_MAXNEWSIZESet Java max new size.
Druid releases & version history
Apache Druid 38.0.0 contains over 250 new features, bug fixes, performance enhancements, documentation improvements, and additional test coverage from 37 contributors.
See the complete set of changes for additional details, including bug fixes.
Review the upgrade notes and incompatible changes before you upgrade to Druid 38.0.0.
If you are upgrading across multiple versions, see the Upgrade notes page, which lists upgrade notes for the most recent Druid versions.
Important features, changes, and deprecations
This section contains important information about new and existing features.
Java
Druid now supports Java 25. While Java 21 is still supported, we recommend you upgrade to Java 25.
Support for Java 17 has been dropped.
Historical tier aliases
You can map a virtual tier name to a number of real Historical tiers with the historicalTierAliases Coordinator dynamic configuration. This creates a single identifier for a group of Historical tiers. When a load/drop rule references the alias, the Coordinator replaces it with the actual tiers. For example, if you map the Historical tiers hot_1 and hot_2 to the alias hot, the rule {"hot": 2} loads 2 replicas of each onto hot_1 and hot_2.
Realtime segments mode query context
The realtimeSegmentsOnly query context parameter has been deprecated and replaced with realtimeSegmentsMode.
You can set realtimeSegmentsMode to one of the following:
include(default): query all segments, including realtime segmentsexclude: skip realtime segments for the queryexclusive: query only realtime segments
This is useful when performing things like blue/green deployments and you only want to query new Historical replica autoscaling groups and not touch any "live" nodes (neither realtime nor historical).
Faster segment metadata search
You can set druid.segment.timeline.fastIntervalSearch to true so that Druid uses an index based on interval trees to store that metadata in memory for faster identification and retrieval.
This feature is off by default.
Fetching files in the background
The MSQ task engine now supports workers fetching input files from cloud storage asynchronously. This overlaps downloading with processing, which generally improves throughput when reading larger numbers of files. Otherwise, workers stream each file directly from cloud storage while processing it.
This feature is on by default and is controlled by the backgroundFetchExternalFiles setting.
Segment prefetching for Dart
Dart now supports the runtime property druid.msq.dart.worker.segmentLoadAheadCount, which controls the number of segments that Dart prefetches. If set greater than 0 for a worker, this setting becomes the default segmentLoadAheadCount value for the worker. If a query includes the segmentLoadAheadCount query context parameter, the query context takes precedence.
Improved Convert to SQL in web console
The web console now supports converting streaming supervisors to SQL-based ingestion queries. This makes it easier for you to run a streaming task as a one-time batch ingestion.
Select the Convert supervisor to SQL option from the ... menu in the Query view. You can select from existing supervisors or provide JSON for a new one.
Deprecate legacy ZSTD frames
Druid 38 will be the last major Druid release that supports reading from legacy v0.4-v0.7 ZSTD frames. Support will be removed for these frames in future releases for both batch and streaming ingestion.
The primary reason for this deprecation is that the underlying library zstd-jni removes decoding for legacy frames in version 1.5.7-13 onward.
Functional area and related changes
This section contains detailed release notes separated by areas.
Web console
Other web console improvements
- Added the following status details to the Services view for Historical services: cloning from another Historical, in turbo loading mode, in decommissioning mode #19253
- Added support for resetting a supervisor to the latest offsets and backfilling #19533
- Improved handling for new tabs #19483
- Improved the Home view's Services card. It now reports Overlord, Coordinator, Router, Broker, and Indexer counts on clusters where the web console talks to the Coordinator without SQL access #19481
Ingestion
- You can now use the expression aggregator at ingestion time for expressions that produce a LONG or DOUBLE for both fold and combine expressions #19508
- Added
now()expression function that returns the current system timestamp in milliseconds since epoch. Useful at ingestion time for troubleshooting pipeline delays (e.g.,now() - __time). Note:now()is non-deterministic as it evaluates for every row, so it can break idempotency. This can be added to any column besides__time#19386 - Improved resiliency when ingesting from S3. Druid now retries on
SSLExceptionand transient credential errors instead of failing #19617 #19558 - Improved S3 performance #19394
- Updated the default S3 connection pool size so that it's computed based on the number of available processors #19536
SQL-based ingestion
EXTERN for S3 now supports role ARN
You can now include the role ARN when running an INSERT INTO EXTERN query. For example:
INSERT INTO
EXTERN(
s3(bucket => 'test2', prefix => 'export', assumeRoleArn => 'arn:aws:iam::00000:role/test-20260520'))
AS CSV
SELECT ...
Other SQL-based ingestion improvements
- Added storage counters for the amount of bytes and files written to local and durable storage as well as the state of the local
ByteTracker. Per-worker storage counters have also been added to the web console #19316
Streaming ingestion
Scaling cool down
You can now configure different cool down durations for scaling up and scaling down streaming task autoscalers.
Improved supervisor restarts
Supervisors no longer restart for all changes. Based on the type of change, one of the following can occur:
- Druid persists the updated spec without a restart.
- Druid restarts the supervisor, but running tasks aren't impacted.
- Druid restarts the supervisor is restarted and its tasks are terminated, which was the default behavior before this change.
For example, cosmetic changes to a supervisor spec no longer trigger a restart.
Additionally, the algorithm for determining a change in the spec has been improved. For example, changes to ioConfig.taskCount don't trigger a supervisor restart if auto-scaling is enabled.
Latest offset and backfill
For Kafka and RabbitMQ, you can now reset a supervisor to the latest offset and start a new bounded backfill supervisor to ingest data from the skipped range. This is a useful feature for operating Druid clusters where the most recent data is the most important, like alerting scenarios.
Note the following requirements:
- The supervisor's
useEarliestSequenceNumberproperty must befalse. - The supervisor context must have
useConcurrentLocksset totrueto allow the backfill supervisor's tasks to write concurrently with the main supervisor's tasks. - The supervisor must be in a
RUNNINGstate.
Use POST /druid/indexer/v1/supervisor/{supervisorId}/resetToLatestAndBackfill or the web console to perform this action.
Prunable shard specs for streaming published segments
Kafka ingestion can now publish segments that the Broker prunes at query time without waiting for compaction. Set tuningConfig.streamingPartitionsSpec.partitionDimensions to a list of low-to-medium cardinality dimensions; each task records
the distinct values it observes per dimension and stamps them onto a new dim_value_set shard spec. Queries that filter on a declared dimension then skip segments whose values can't match.
The feature is opt-in, Kafka-only, and disabled by
default; when unset, behavior is unchanged.
dim_value_set is a new core shard spec type with no fallback, so it is not forward-compatible. Upgrade all services before enabling streamingPartitionsSpec. Once dim_value_set segments are published, downgrade is unsupported
until they are compacted away or streamingPartitionsSpec is removed.
Added maxValuesPerDimension (optional) to streamingPartitionsSpec.
Other streaming ingestion improvements
- Added a property called
boundedStreamConfigto theSeekableStreamSupervisorIOConfig, which allows operators to spin up a Supervisor that consumes only a specified offset range #19372 - Improved the cost-based autoscaler for better throughput #19646
Querying
New query laning strategy
The weighted query laning strategy scores queries by how many thresholds they breach (segment count, interval duration, data age, segment range) and assigns them to configurable graduated lanes with different capacity limits, providing more nuanced lane assignment than the existing binary high/low strategy.
The weighted query laning strategy supports optional per-threshold cost weights:
periodWeightdurationWeightsegmentCountWeightsegmentRangeWeight
These threshold weights default to 1.
Other querying improvements
- Added nullable
minTime/maxTimeLong fields toProjectionMetadata#19398 - Added
getDimensionRangeSetsupport toLikeDimFilterfor equality and prefix cases #19524 - Added support for aggregate projections with clustered segments #19599
- Optimized performance of aggregators for groupBy queries #19423
Cluster management
Improved diskNormalized balancer strategy
The diskNormalized strategy is now more tunable. The utilizationThreshold configuration works more intuitively: increasing the threshold increases the "tolerance" of the strategy while decreasing the threshold pushes nodes' disk utilization closer together.
MiddleManager and Indexer restarts
MiddleManagers and Indexers now persist their enabled or disabled state across restarts. If you prefer the old behavior, where the server re-enables itself after a restart, set druid.worker.startAlwaysEnabled = true.
Kafka idle signal
Improved the cost-based auto scaler for Kafka. The poll-idle ratio only reflected the time spent polling, whether there is spare processing capacity. You can now configure the autoscaler to use a utilization ratio instead:
1 - (avgProcessingRate / maxObservedRate)
Set useUtilizationRatio to true to use this new ratio for autoscaling.
Other cluster management improvements
- Added
datasourcefilter pushdown tosys.segmentstable #19718 #19731 - Added a
restartedboolean field to the supervisor POST endpoint response to indicate whether the supervisor was actually restarted #19349 - Added
error_messagecolumn tosys.server_propertiestable and made the table resilient to unreachable servers. Previously, the entire query would fail if any server was unreachable; now a row is returned witherror_messagepopulated. The table also now supports filter and projection pushdown #19459 - Added
druid.expressions.useVectorApiconfig to support the incubating JDK Vector API. To use the API, set the config totrueand start Druid with the--add-modules=jdk.incubator.vectorflag #19512 - Added debug logging at the INFO level for projections if the debug flag is set #19613
- The MSQ task engine now logs the full stack trace when
debugis set in the context #19361 - Improved the cost-based autoscaler so that it scales down over-provisioned supervisors running above the ideal idle ratio with low lag #19562
- Improved how Druid handles Java. MiddleManagers now honor
JAVA_HOME#19709
Data management
Other data management improvements
- Improved how compaction supervisor specs start up. They no longer attempt to create tasks with invalid configs #19223
- Changed Historical tiers so that they can only be associated with one tier alias #19595
- Sped up segment metadata cache syncs #19672
Metrics and monitoring
New metadata cache metrics
Added the following metrics for the segment metadata cache:
segment/metadataCache/unused/countsegment/metadataCache/fetchIds/timesegment/metadataCache/fetchPayloads/timesegment/metadataCache/fetchPending/timesegment/metadataCache/fetchSchemas/timesegment/metadataCache/fetchIndexingStates/timesegment/metadataCache/updateIds/timesegment/metadataCache/updateSnapshot/timesegment/metadataCache/schema/skippedsegment/metadataCache/indexingState/addedsegment/metadataCache/indexingState/deleted
Changed metrics for cost-based autoscaling
Removed the following metrics:
task/autoScaler/costBased/lagCosttask/autoScaler/costBased/idleCost
Added the following metrics:
task/autoScaler/costBased/avgProcessingRatetask/autoScaler/costBased/avgPollIdleRatiotask/autoScaler/costBased/lagWeighttask/autoScaler/costBased/costWeight
Auth metrics
You can now configure Druid to emit metrics for authorization events.
The auth/forbidden and auth/exception metrics have the following dimensions to support precise alerting on security events:
identityauthorizerNameresourceNameresourceTypeactionerrorMessagewhere applicable
To enable this functionality, set druid.auth.emitAuthMetrics to true.
Storage metrics
The storage/load/bytes and storage/virtual/load/bytes metrics now measure once the load is complete. Previously, they measured when the load starts.
Additionally, storage/load/begin/bytes and storage/virtual/load/begin/bytes have been introduced and have the previous function of storage/load/bytes and storage/virtual/load/bytes.
The count metrics have also been updated to reflect this.
Concurrent append and replace
Added the following metrics for concurrent append and replace in realtime ingestion tasks:
ingest/segmentUpgrade/countingest/segmentUpgrade/notifiedingest/segmentUpgrade/unmatchedingest/segmentUpgrade/sendFailedingest/segmentUpgrade/announcedingest/segmentUpgrade/skipped
Other metrics and monitoring improvements
- Added the metric
segment/allocated/countto track IDs of allocated pending segments #19674 - Added
remoteAddressdimension for JDBC/Avatica queries to the following metrics:sqlQuery/time,sqlQuery/bytes, andsqlQuery/planningTimeMs#19231 - Added
identitydimension onquery/timemetric for the Router #19342 - Added
kafka/consumer/pollIdleRatio, which corresponds to the Kafka consumerpoll-idle-ratio-avg#19366 - Added
supervisorIdto Kafka consumer metrics #19525 - Added
tierAliasdimension to some tiered metrics, making it easier to aggregate across aliases for the monitoring and alerting #19595 - Added the
query/segments/countmetric to data nodes #19624 - Added the following metrics for virtual storage:
storage/virtual/read/count,storage/virtual/read/bytes,storage/virtual/read/time#19632 - Changed
query/node/{bytes/time}and backpressure metrics to emit even on query failure to data nodes #19453 - Improved metrics for partial segment loading #19632
- Improved router logging to always include
statusCodedimension #19668
Extensions
Redis
You can now enable TLS support for Redis connections.
pac4j OIDC authentication
Users of the druid-pac4j OIDC authentication extension can now explicitly configure their preferred client authentication method using the new optional clientAuthenticationMethod parameter. This resolves compatibility issues introduced with pac4j 5.7.3 where OIDC providers advertising private_key_jwt would cause authentication failures when the asymmetric JWT method was not configured.
Supported values include: client_secret_basic, client_secret_post, client_secret_jwt, private_key_jwt, and none. If not specified, pac4j will continue to use its auto-detection behavior.
Kubernetes
- Added
sysmetrics for k8s peons #19305 - Added experimental support for running Kubernetes indexing tasks across multiple Kubernetes clusters. Set
druid.indexer.runner.type=multik8sand configuredruid.indexer.runner.clustersto schedule tasks across multiple Kubernetes clusters from a single Overlord #19433 jvm.configConfigMaps are now honored by Peons #19364- You can now set
podTemplateSelectionKeyin a task's context to pick a specific configured pod template (e.g.druid.indexer.runner.k8s.podTemplate.<selectionKey>) without configuring aselectorBasedstrategy. The override takes precedence over both the default andselectorBasedstrategies. The feature is controlled by the new runtime propertydruid.indexer.runner.allowTaskPodTemplateSelection, which defaults tofalse. If the named template isn't configured, the task fails to launch. #19419
T-digest sketch
You can now pin the maximum available compression on T-digest operators to bind resources using the druid.tdigest.maxCompression parameter.
OpenLineage
Added extensions-contrib/openlineage-emitter as a community extension. It uses the RequestLogger to transform and send lineage information to any OpenLineage-compatible API.
Upgrade notes and incompatible changes
Upgrade notes
Java
Druid now supports Java 25. While Java 21 is still supported, we recommend you upgrade to Java 25.
Support for Java 17 has been dropped.
JAVA_HOME
When Druid uses the bundled bin/run-java script during startup, it honors the DRUID_JAVA_HOME/JAVA_HOME environment variables. Otherwise, Druid falls back to java on the PATH.
ZooKeeper-based task runner
The ZooKeeper-based RemoteTaskRunner (druid.indexer.runner.type=remote) has been removed. The HTTP-based httpRemote runner has been the default since Druid 25.0.0 and is now the only supported distributed task runner. local (in-process) remains supported for single-process testing, and the k8s runner from the Kubernetes extension is unaffected.
If your configuration sets druid.indexer.runner.type=remote, startup fails. Remove the property or set it to httpRemote (which is the default) to proceed.
The following configuration properties are no longer recognized and should be removed from common.runtime.properties:
druid.indexer.runner.maxZnodeBytesdruid.indexer.runner.taskShutdownLinkTimeoutdruid.indexer.runner.compressZnodesdruid.zk.paths.indexer.basedruid.zk.paths.indexer.announcementsPathdruid.zk.paths.indexer.tasksPathdruid.zk.paths.indexer.statusPath
ZooKeeper is still used for Coordinator/Overlord leader election and service (node) announcement and discovery.
ZooKeeper-based segment announcement and discovery removed
The ZooKeeper-based segment announcement and inventory view, which have been deprecated and off by default for several releases, have been removed. The HTTP-based path (the default for druid.serverview.type=http) is now the only supported option.
If your configuration sets druid.serverview.type to anything other than http, startup now fails with a clear error message. Remove the property (or set it to http, which is the default) to proceed.
The following configuration properties are no longer recognized and should be removed from common.runtime.properties:
druid.announcer.segmentsPerNodedruid.announcer.maxBytesPerNodedruid.announcer.skipLoadSpecdruid.announcer.skipDimensionsAndMetricsdruid.announcer.skipSegmentAnnouncementOnZkdruid.zk.paths.announcementsPathdruid.zk.paths.liveSegmentsPathdruid.zk.paths.propertiesPathdruid.zk.paths.connectorPath
ZooKeeper is still used for leader election, service (node) announcement and discovery, and Overlord-to-MiddleManager task management.
Jackson update
If external code has both @JacksonInject and @JsonProperty on the same parameter and relies on the JSON value winning when supplied, add the explicit useInput = OptBoolean.TRUE to the annotation (or stay on Jackson 2.20.x). All such sites in Druid itself have been updated.
Incompatible changes
Java 17
Support for Java 17 has been dropped. #19304
Developer notes
- Changed the shard spec collector to an interface to make extensibility easier #19744
- Changed
iceberg.core.versionin root pom and aligned embedded tests to use1.10.0 - Updated tests in
druid-processingto use JUnit 5 #19601 - Updated the
apache/druidandapache/druid-website-srcrepositories to use Docusaurus 3.10 when building the website #19522 - Updated IntelliJ settings to match Druid's supported Java versions #19661
Dependency updates
The following dependencies have had their versions bumped:
org.apache.logging.log4jfrom2.25.4to2.26.0#19629com.fasterxml.jacksonfrom2.21.3to2.21.4#19618io.nettyfrom4.2.12.Finalto4.2.15.Final#19566caffeinefrom2.9.3to3.2.4#19527errorpronefrom2.41.0to2.49.0#19527jacksonfrom2.20.2to2.21.3#19528derbyfrom10.14.2.0to10.17.1.0#19492postgresfrom42.7.2to42.7.11#19474org.bouncycastlefrom1.82to1.84#19473
- Apache Kafka client and broker dependencies from
3.9.2to4.2.3#19441 #19584 axiosfrom 1.15.0 to 1.15.2 #19430pac4jfrom5.7.3to5.7.10#19388jose.jwtfrom9.37.2to9.37.3#19388log4jfrom2.25.3to2.25.4#19388RoaringBitmapfrom1.6.13to1.6.14#19688
Credits
@317brian
@abhishekrb19
@adarshsanjeev
@Adithya-Shetty100
@aho135
@amaechler
@ashwintumma23
@avaamsel
@beenhead
@capistrant
@cecemei
@clintropolis
@cryptoe
@dependabot[bot]
@dervoeti
@EdwinIngJ
@Fly-Style
@FrankChen021
@gianm
@github-actions[bot]
@GWphua
@jaykanakiya
@jtuglu1
@JWuCines
@kfaraz
@LakshSingla
@maytasm
@mshahid6
@MukundaKatta
@rinchinov
@rmn7
@Shekharrajak
@vogievetsky
@vtlim
@winsmith
@wlswo
@Yomanz
Apache Druid 37.0.0 contains over 255 new features, bug fixes, performance enhancements, documentation improvements, and additional test coverage from 29 contributors.
See the complete set of changes for additional details, including bug fixes.
Review the upgrade notes and incompatible changes before you upgrade to Druid 37.0.0.
If you are upgrading across multiple versions, see the Upgrade notes page, which lists upgrade notes for the most recent Druid versions.
Important features, changes, and deprecations
This section contains important information about new and existing features.
Hadoop-based ingestion
Support for Hadoop-based ingestion has been removed. The feature was deprecated in Druid 34.
Use one of Druid's other supported ingestion methods, such as SQL-based ingestion or MiddleManager-less ingestion using Kubernetes.
Query blocklist
You can now use the Broker API (/druid/coordinator/v1/config/broker) to create a query blocklist to dynamically block queries by datasource, query type, or query context. The blocklist takes effect without a restarting Druid. Block rules use AND logic, which means all criteria must match.
The following example blocks all groupBy queries on the wikipedia datasource with a query context parameter of priority equal to 0:
POST /druid/coordinator/v1/config/broker
{
"queryBlocklist": [
{
"ruleName": "block-wikipedia-groupbys",
"dataSources": ["wikipedia"],
"queryTypes": ["groupBy"],
"contextMatches": {"priority": "0"}
}
]
}
Minor compaction for Overlord-based compaction (experimental)
You can now configure minor compaction to compact only newly ingested segments while upgrading existing compacted segments. When Druid upgrades segments, it updates the metadata instead of using resources to compact it again. You can use the native compaction engine or the MSQ task engine.
Use the mostFragmentedFirst compaction policy and set either a percentage of rows-based or byte-based threshold for minor compaction.
Cascading reindexing (experimental)
Using cascading reindexing, you can now define age-based rules to automatically apply different compaction configurations based on the age of your data. While standard auto-compaction applies a single flat configuration across an entire datasource, cascading reindexing lets you tailor your compaction settings to the characteristics of your data.
For example, you can keep recent data in hourly segments while automatically rolling up to daily segments after 90 days to reduce segment count. You can also layer on age-based row deletion (such as dropping bot traffic from older data), change compression settings, or shift to rollup with coarser query granularity as data ages. Rules are defined inline in the supervisor spec.
You must use compaction supervisors with the MSQ task engine to use cascading reindexing.
Multi-supervisor ingestion
Multi-supervisor ingestion is now generally available. You can run multiple stream supervisors that ingest into the same datasource.
Read-only authorizer
Added a ReadOnly authorizer to Druid. This is the first global authorizer for Druid. The authorizer enforces a global restriction on all non-READ operations, denying them regardless of individual user permissions. You can use this capability to ensure all users of a specific authorizer are limited to READ access.
There is a known limitation where some endpoints currently require WRITE access despite being READ-only, such as GET /druid/indexer/v1/supervisor. These operations will fail.
Thrift input format
As part of the Thrift contributor extension, Druid now supports Thrift-encoded data for Kafka and Kinesis streaming ingestion using InputFormat. Previously, Druid supported this through parsers, which have been removed in Druid 37.
To use this feature, you must add druid-thrift-extensions to your extension load list.
Incremental cache
Incremental segment metadata cache (useIncrementalCache) is now generally available and defaults to ifSynced. Druid blocks reads from the cache until it has synced with the metadata store at least once after becoming leader.
Kubernetes-based task management
This extension is now generally available.
Dynamic default query context
You can now add default query context parameters as a dynamic configuration to the Broker. This allows you to override static defaults set in your runtime properties without restarting your deployment or having to update multiple queries individually. Druid applies query context parameters based on the following priority:
- The query context included with the query
- The query context set as a dynamic configuration on the Broker
- The query context parameters set in the runtime properties
- The defaults that ship with Druid
Note that like other Broker dynamic configuration, this is best-effort. Settings may not be applied in certain
cases, such as when a Broker has recently started and hasn't received the configuration yet, or if the
Broker can't contact the Coordinator. If a query context parameter is critical for all your queries, set it in the runtime properties.
sys.queries table (experimental)
The new system queries table provides information about currently running and recently completed queries that use the Dart engine. This table is off by default. To enable the table, set the following:
druid.sql.planner.enableSysQueriesTable = true
As part of this change, the /druid/v2/sql/queries API now supports an includeComplete parameter that shows recently completed queries.
Auto-compaction with compaction supervisors
Auto-compaction using compaction supervisors has been improved, now generally available, and the recommended default. Automatic compaction tasks are now prefixed with auto instead of coordinator-issued.
As part of the improvement compaction states are now stored in a central location, a new indexingStates table. Individual segments only need to store a unique reference (indexing_state_fingerprint) to their full compaction state.
Since many segments in a single datasource share the same underlying compaction state, this greatly reduces metadata storage requirements for automatic compaction.
For backwards compatibility, Druid continues to persist the detailed compaction state in each segment. This functionality will be removed in a future release.
You can stop storing detailed compaction state by setting storeCompactionStatePerSegment to false in the cluster compaction config. If you turn it off and need to downgrade, Druid needs to re-compact any segments that have been compacted since you changed the config.
This change has upgrade impacts for metadata storage and metadata caching. For more information, see the Metadata storage for auto-compaction with compaction supervisors upgrade note.
Broker tier selection for realtime servers
Added druid.broker.realtime.select.tier and druid.broker.realtime.balancer.type on the Brokers to optionally override the Broker’s tier selection and balancer strategies for realtime servers. If these properties are not set (the default), realtime servers continue to use the existing druid.broker.select and druid.broker.balancer configurations that apply to both historical and realtime servers.
Manual Broker routing in the web console
You can now configure which Broker the Router uses for queries issued from the web console. You may want to do this if there are Brokers that don't have visibility into certain data tiers, and you know you're querying data available only on a certain tier.
To specify a Broker, add the following config to web-console/console-config.js:
consoleBrokerService: 'druid/BROKER_NAME'Consul extension
The contributor extension druid-consul-extensions lets Druid clusters use Consul for service discovery and
Coordinator/Overlord leader election instead of ZooKeeper. The extension supports ACLs, TLS/mTLS, and metrics.
Before you switch to Consul, you need to set
druid.serverview.type=http and druid.indexer.runner.type=httpRemote cluster wide.
Functional area and related changes
This section contains detailed release notes separated by areas.
Web console
Changed storage column displays
The following improvements have been made to how storage columns are displayed in the web console:
- Improved the compaction config view to
- Renamed Current size to Assigned size.
- Renamed Max size to Effective size. It now displays the smaller value between
max_sizeandstorage_size. The max size is still shown as a tooltip. - Changed usage calculation to use
effective_size
Other web console improvements
- Added
workerDesctoWorkerStats, which makes it easier to identify where a worker is running #19171 - Added the Dart unique execution ID (
dartQueryId) and thesqlQueryIdto the Details pane in the web console #19185 - Added support for showing completed Dart queries in the web console #18940
- Added a detail dialog to the Services page #18960
- Added icons to indicate when data is loaded into virtual storage, including a tooltip that shows all the counters for the data #19010
- Added support for Dart reports #18897
- Changed the criteria for active workers: any nonzero rows, files, bytes, frames, or wall time is enough to consider a worker active #19183
- Changed the Cancel query option to show only if a query is in an accepted or running state #19182
- Changed the ordering of the current Dart queries panel to show queries in the following order: RUNNING, ACCEPTED, and then COMPLETED. RUNNING and ACCEPTED queries are ordered by the most recent first (based on timestamp). COMPLETED queries are sorted by finish time #19237
Ingestion
Truncate string columns
Use the StringColumnFormatSpec config to set the maximum length for string dimension columns you ingest:
- For a specific dimension:
dimensionSchema.columnFormatSpec.maxStringLength - For a specific job:
indexSpec.columnFormatSpec.maxStringLength - Cluster-wide:
druid.indexing.formats.maxStringLength
Druid truncates any string longer than the specified length. The default is to not truncate string values.
Other ingestion improvements
- Sped up task scheduling on the Overlord #19199
SQL-based ingestion
Other SQL-based ingestion improvements
- Added support for virtual storage fabric when performing SQL-based ingestion #18873
- Added support for
StorageMonitorso that MSQ task engine tasks always emittaskIdandgroupId#19048 - Improved worker cancellation #18931
- Improved exception handling #19234
Streaming ingestion
Changed how tasks get launched for autoscaling
The behavior of taskCountMin and taskCountStart has been changed for autoscaling. Druid now computes the initial number of tasks to launch by checking the configs in the following order: taskCountStart (optional), then taskCount (in ioConfig), then taskCountMin.
Improved query isolation
Added serverPriorityToReplicas parameter to the streaming supervisor specs (Kafka, Kinesis, and Rabbit). This allows operators to distribute task replicas across different server priorities for realtime indexing tasks. Similar to Historical tiering, this enables query isolation for mixed workload scenarios on the Peons, allowing some task replicas to handle queries of specific priorities.
Other streaming ingestion improvements
- Improved cost-based autoscaler performance in high lag scenarios #19045
- Improved the performance of realtime task scheduling by ordering schedule requests by priority on the
TaskQueue#19203
Querying
groupBy query configuration
Added a new groupBy query configuration property druid.query.groupBy.maxSpillFileCount to limit the maximum number of spill files created per query. When the limit is exceeded, the query fails with a clear error message instead of causing Historical nodes to run out of memory during spill file merging. The limit can also be overridden per query using the query context parameter maxSpillFileCount.
Improved handling of nested aggregates
Druid can now merge two aggregates with a projection between them. For example, the following query:
SELECT hr, UPPER(t1.x) x, SUM(t1.cnt) cnt, MIN(t1.mn) mn, MAX(t1.mx) mx FROM ( SELECT floor(__time to hour) hr, dim2 x, COUNT(*) cnt, MIN(m1 * 5) mn, MAX(m1 + m2) mx FROM druid.foo WHERE dim2 IN ('abc', 'def', 'a', 'b', '') GROUP BY 1, 2 ) t1 WHERE t1.x IN ('abc', 'foo', 'bar', 'a', '') GROUP BY 1, 2
can be simplified to the following:
SELECT FLOOR(__time TO hour) hr, UPPER(dim2) x, COUNT(*) cnt, MIN(m1 * 5) mn, MAX(m1 + m2) mx FROM druid.foo WHERE dim2 IN ('abc', 'a', '') GROUP BY 1, 2
Other querying improvements
- Added
durationMsto Dart query reports #19169 - Improved error handling so that row signature column order is preserved when column analysis encounters an error #19162
- Improved GROUP BY performance #18952
- Improved expression filters to take advantage of specialized virtual columns when possible, resulting in better performance for the query #18965
Cluster management
New Broker tier selection strategies
Operators can now configure two new Broker TierSelectorStrategy implementations:
strict- Only selects servers whose priorities match the configured list. Example configuration:druid.broker.select.tier=strictanddruid.broker.select.tier.strict.priorities=[1].pooled- Pools servers across the configured priorities and selects among them, allowing queries to use multiple priority tiers for improved availability. Example configuration:druid.broker.select.tier=pooledanddruid.broker.select.tier.pooled.priorities=[2,1].
You can also use druid.broker.realtime.select.tier to configure these strategies for realtime servers.
Druid operator
The Druid operator now resides in its own repository: apache/druid-operator.
Cost-based autoscaler algorithm
The algorithm for cost-based autoscaling has been changed:
- Scale up more aggressively when per-partition lag is meaningful
- Relax the partitions-per-task increase limit based on lag severity and headroom
- Keep behavior conservative near
taskCountMaxand avoid negative headroom effects
Other cluster management improvements
- Added
/status/readyendpoint for service health so that external load balancers can handle a graceful shutdown better #19148 - Added a configurable option to scale-down during task run time for cost-based autoscaler #18958
- Added
storage_sizetosys.serversto facilitate retrieving disk cache size for Historicals when using the virtual storage fabric #18979 - Added a log for new task count computation for the cost-based auto scaler #18929
- Changed how the scaling is calculated from a square root-based scaling formula to a logarithmic formula that provides better emergency recovery at low task counts and millions of lag #18976
- Improved the load speed of cached segments during Historical startup #18489
- Improved Broker startup time by parallelizing buffer initialization #19025
- Improved the stack trace for MSQ task engine worker failures so that they're preserved #19049
- Improved the performance of the cost-based autoscaler during loaded lag conditions #18991
Data management
Per-segment timeout configuration
You can now set a timeout for the segments in a specific datasource using a dynamic configuration:
POST /druid/coordinator/v1/config/broker
{
"perSegmentTimeoutConfig": {
"my_large_datasource": { "perSegmentTimeoutMs": 5000, "monitorOnly": false },
"my_new_datasource": { "perSegmentTimeoutMs": 3000, "monitorOnly": true }
}
}
This is useful when different datasources have different performance characteristics — for example, allowing longer timeouts for larger datasets.
Durable storage cleaner
The durable storage cleaner now supports configurable time-based retention for MSQ query results. Previously, query results were retained for all known tasks, which was unreliable for completed tasks. With this change, query results are retained for a configurable time period based on the task creation time.
The new configuration property druid.msq.intermediate.storage.cleaner.durationToRetain controls the retention period for query results. The default retention period is 6 hours.
Other data management improvements
- Added the
druid.storage.transfer.asyncHttpClientTypeconfig that specifies which async HTTP client to use for S3 transfers:crtfor Amazon CRT ornettyfor Netty NIO #19249 - Added a mechanism to automatically clean up intermediary files on HDFS storage #19187
Metrics and monitoring
buildRevision field
All Druid metrics now include a buildRevision field to help identify the Git build revision of the Druid server emitting a metric. You can use this information to verify that all nodes in a cluster are running the intended revision.
Monitoring supervisor state
Added a new supervisor/count metric when SupervisorStatsMonitor is enabled in druid.monitoring.monitors. The metric reports each supervisor’s state, such as RUNNING or SUSPENDED, for Prometheus, StatsD, and other metric systems.
Improved groupBy metrics
GroupByStatsMonitor now provides the following metrics:
mergeBuffer/bytesUsedmergeBuffer/maxBytesUsedmergeBuffer/maxAcquisitionTimeNsgroupBy/maxSpilledBytesgroupBy/maxMergeDictionarySize
Filtering metrics
Operators can set druid.emitter.logging.shouldFilterMetrics=true to limit which metrics the logging emitter writes. Optionally, they can set druid.emitter.logging.allowedMetricsPath to a JSON object file where the keys are metric names. A missing custom file results in a warning and use of the bundled loggingEmitterAllowedMetrics.json. Alerts and other non-metric events are always logged.
New Broker metrics
Added segment/schemaCache/rowSignature/changed and segment/schemaCache/rowSignature/column/count metrics to expose events when the Broker initializes and updates the row signature in the segment metadata cache for each datasource.
Other metrics and monitoring improvements
- Added the following metrics to the default for Prometheus:
mergeBuffer/bytesUsedandmergeBuffer/maxBytesUsed#19110 - Added compaction mode to the
compact/task/countmetric #19151 - Added support for logging and emitting SQL dynamic parameter values #19067
- Added
ingest/rows/published, which all task types emit to denote the total row count of successfully published segments #19177 - Added
queriesandtotalQueriescounters, which reflect queries made to realtime servers to retrieve realtime data #19196 - Added
tier/storage/capacitymetric for the Coordinator. This metric is guaranteed to reflect the totalStorageLocationsize configured across all Historicals in a tier #18962 - Added new metrics for virtual storage fabric to the MSQ task engine
ChannelCounters:loadBytes,loadTime,loadWait, andloadFiles#18971 - Added
storage/virtual/hit/bytes,storage/virtual/hold/countandstorage/virtual/hold/bytesmetric toStorageMonitor
#18895 #19217 - Added
supervisorIddimension for streaming tasks toTaskCountStatsMonitor#18920 - Changed
StorageMonitorto always be on #19048 - Improved the metrics for autoscalers, so that they all emit the same metrics:
supervisorId,dataSource, andstream#19097
Extensions
Kubernetes
Added a new WebClientOptions pass-through for the Vert.x HTTP client in the kubernetes-overlord-extensions. Operators can now configure any property on the underlying Vert.x WebClientOptions object by using Druid runtime properties. Some of the options you can configure include connection pool size, keep-alive timeouts, and idle timeouts. This is particularly useful for environments with intermediate load balancers that close idle connections. Most Druid deployments will not need this configuration.
gRPC
The gRPC query extension now cancels in-flight queries when clients cancel or disconnect.
Iceberg
GCS warehouse
The Iceberg input source now supports GCS warehouses. To use this feature, you must load the druid-google-extensions extension in addition to the Iceberg extension.
#19137
Filters
You can now configure residual filters for non-partition columns when using the Iceberg input source. Set residualFilterMode in the Iceberg input source to one of the following:
ignore: (default) Ingest the residual rows with a warning log unless filtered bytransformSpec.fail: Fail the ingestion job when residual filters are detected. Use this to ensure that filters only target partition columns.
HDFS storage
Added support for lz4 compression. As part of this change, the following metrics are now available:
hdfs/pull/sizehdfs/pull/durationhdfs/push/sizehdfs/push/duration
Upgrade notes and incompatible changes
Upgrade notes
Hadoop-based ingestion
Support for Hadoop-based ingestion has been removed. The feature was deprecated in Druid 34.
Use one of Druid's other supported ingestion methods, such as SQL-based ingestion or MiddleManager-less ingestion using Kubernetes.
AWS SDK v2
Druid now uses AWS SDK version 2.40.0 since v1 of the SDK is at end of life.
Segment metadata cache on by default
Starting in Druid 37, the segment metadata cache is on by default. This feature allows the Broker to cache segment metadata polled from the Coordinator, rather than having to fetch metadata for every query against the sys.segments table. This improves performance but increases memory usage on Brokers.
The druid.sql.planner.metadataSegmentCacheEnable config controls this feature.
Parser changes
Streaming ingestion parser
Support for the deprecated parser has been removed for streaming ingest tasks such as Kafka and Kinesis. Operators must now specify inputSource/inputFormat on the ioConfig of the supervisor spec, and the dataSchema must not specify a parser. Do this before upgrading to Druid 37 or newer.
Removed ParseSpec and deprecated parsers
The Parser for native batch tasks and streaming ingestion indexing services has been removed. Where possible, use the input format instead. Note that JavascriptParseSpec and JSONLowercaseParseSpec have no InputFormat equivalents.
Druid supports custom text data formats and can use the Regex input format to parse them. However, be aware doing this to
parse data is less efficient than writing a native Java InputFormat extension, or using an external stream processor. We welcome contributions of new input formats.
Rolling upgrades from Druid versions prior to version 0.23
You can't perform a rolling upgrade from versions earlier than Druid 0.23.
Metadata storage for auto-compaction with compaction supervisors
Automatic compaction with supervisors requires incremental segment metadata caching on the Overlord and a new metadata store table; no action is required if you are using the default settings for the following configs:
druid.manager.segments.useIncrementalCachedruid.metadata.storage.connector.createTables
If druid.manager.segments.useIncrementalCache is set to never, update it to ifSynced or always. For more information about the config, see Segment metadata cache.
If you set the druid.metadata.storage.connector.createTables config to false, you need to manually alter the segments table and create the compactionStates table. The Postgres DDL is provided below as a guide:
-- create the indexing states lookup table and associated indices
CREATE TABLE druid_indexingStates (
created_date VARCHAR(255) NOT NULL,
datasource VARCHAR(255) NOT NULL,
fingerprint VARCHAR(255) NOT NULL,
payload BYTEA NOT NULL,
used BOOLEAN NOT NULL,
pending BOOLEAN NOT NULL,
used_status_last_updated VARCHAR(255) NOT NULL,
PRIMARY KEY (fingerprint),
);
CREATE INDEX idx_druid_compactionStates_used ON druid_compactionStates(used, used_status_last_updated);
-- modify druid_segments table to have a column for storing compaction state fingerprints
ALTER TABLE druid_segments ADD COLUMN indexing_state_fingerprint VARCHAR(255);
You may have to adapt the syntax to fit your table naming prefix and metadata store backend.
Segment locking
Segment locking and NumberedOverwriteShardSpec are deprecated and will be removed in a future release. Use time chunk locking instead. You can make sure only time chunk locking is used by setting druid.indexer.tasklock.forceTimeChunkLock to true, which is the default.
Incompatible changes
Removed defaultProcessingRate config
This config allowed scaling actions to begin prior to the first metrics becoming available.
Front-coding format
Druid now defaults to v1 of the front-coded format instead of version 0 if enabled. Version 1 was introduced in Druid 26. Downgrading to or upgrading from a version of Druid prior to 26 may require reindexing if you have front-coding enabled with version 0.
Developer notes
- Added
typechecktonpm run test-unitto ensure TypeScript type checking happens on calls totest-unit#19251 - Added a 14 day cooldown to
dependabotupdates to protect against not-yet-discovered regressions and security issues #19241 - Added
AGENTS.md#19084 - Added a requirement to use conventional commit syntax #19089
- Updated
checkstylefrom3.0.0to3.6.0#19064
Dependency updates
The following dependencies have been updated:
- Added
software.amazon.awssdkto supportWebIdentityTokenProvider#19178 org.apache.icebergfrom1.6.1to1.7.2#19172diffnode module from 4.0.1 to 4.0.4 #18933org.apache.avrofrom 1.11.4 to 1.11.5 #19103bytebuddyfrom1.17.7to1.18.3#19000slf4jfrom2.0.16to2.0.17#18990- Apache Commons Codec from
1.16.1to1.17.1#18990 jacocofrom0.8.12to0.8.14#18990docker-java-bomfrom3.6.0to3.7.0#18990assertj-corefrom3.24.2to3.27.7#18994maven-surefire-pluginfrom3.2.5to3.5.4#18847guicefrom5.1.0to6.0.0#18986- JDK compiler from 11 to 17 #18977
vertxfrom4.5.14to4.5.24#18947fabric8from7.4.0to7.5.2#18947mockitofrom5.14.2to5.23#19145easymockfrom5.2.0to5.6.0#19145equalsverifierfrom3.15.8to4.4.1#19145bytebuddyfrom1.18.3to1.18.5#19145- Added
objenesis3.5#19145 org.apache.zookeeperfrom 3.8.4 to 3.8.6 #19135com.lmax.disruptorfrom3.3.6to3.4.4#19122org.junit.junit-bomfrom5.13.3to5.14.3#19122io.fabric8:kubernetes-client7.5.2 → 7.6.0 #19071io.kubernetes:client-java19.0.0 → 25.0.0-legacy #19071com.squareup.okhttp3:okhttp4.12.0 → 5.3.2 #19071org.jetbrains.kotlin:kotlin-stdlib1.9.25 → 2.2.21 #19071commons-codec:commons-codec1.17.1 → 1.20.0 #19071org.apache.commons:commons-lang33.19.0 → 3.20.0 #19071com.google.code.gson:gson2.12.0 → 2.13.2 #19071com.amazonaws:aws-java-sdk1.12.784 → 1.12.793 #19071caffeinefrom2.8.0to2.9.3#19208commons-iofrom2.17.0to2.21.0#19208commons-collections4from4.2to4.5.0#19208commons-compressfrom1.27.0to1.28.0#19208zstd-jnifrom1.5.2-3to1.5.7-7#19208scala-libraryfrom2.12.7to2.13.16#19208icebergfrom1.7.2to1.10.0#19232parquetfrom1.15.2to1.16.0#19232avrofrom1.11.5to1.12.0#19232jacksonfrom2.19.2to2.20.2#19248netty4from4.2.6.Finalto4.2.12.Final#19248errorpronefrom2.35.1to2.41.0#19248bcprov-jdk18on/bcpkix-jdk18onfrom1.81to1.82#19248RoaringBitmapfrom0.9.49to1.6.13#19248jedisfrom5.1.2to7.0.0#19248snakeyamlfrom2.4to2.5#19248aircompressorfrom0.21to2.0.2#19248reflectionsfrom0.9.12to0.10.2#19248httpclient5from5.5to5.5.1#19248jakarta.activationfrom1.2.2to2.0.1#19248netty-tcnative-boringssl-staticfrom2.0.73.Finalto2.0.75.Final#19248maven-compiler-pluginfrom3.11.0to3.14.1#19248
Credits
@317brian
@abhishekrb19
@aho135
@Akshat-Jain
@amaechler
@aruraghuwanshi
@ashwintumma23
@bsmithgall
@capistrant
@cecemei
@clintropolis
@Fly-Style
@FrankChen021
@gianm
@GWphua
@jaykanakiya
@jtuglu1
@kfaraz
@kgyrtkirk
@maytasm
@mshahid6
@puzpuzpuz
@razinbouzar
@santosh-d3vpl3x
@sarangv
@vogievetsky
@vtlim
@wuguowei1994
@zhan7236
Apache Druid 36.0.0 contains over 189 new features, bug fixes, performance enhancements, documentation improvements, and additional test coverage from 34 contributors.
See the complete set of changes for additional details, including bug fixes.
Review the upgrade notes before you upgrade to Druid 36.0.0.
If you are upgrading across multiple versions, see the Upgrade notes page, which lists upgrade notes for the most recent Druid versions.
Important features, changes, and deprecations
This section contains important information about new and existing features.
Functional area and related changes
This section contains detailed release notes separated by areas.
Druid operator
Druid Operator is a Kubernetes controller that manages the lifecycle of your Druid clusters. The operator simplifies the management of Druid clusters with its custom logic that is configurable through
Kubernetes CRDs.
Cost-based autoscaling for streaming ingestion
Druid now supports cost-based autoscaling for streaming ingestion that optimizes task count by balancing lag reduction against resource efficiency.. This autoscaling strategy uses the following formula:
totalCost = lagWeight × lagRecoveryTime + idleWeight × idlenessCost
which accounts for the time to clear the backlog and compute time:
lagRecoveryTime = aggregateLag / (taskCount × avgProcessingRate) — time to clear backlog
idlenessCost = taskCount × taskDuration × predictedIdleRatio — wasted compute time
Kubernetes client mode (experimental)
In kubernetes-overlord-extensions an experimental Kubernetes client mode was added. The new mode uses the fabric8 SharedInformers to cache k8s metadata. This greatly reduces API traffic between the Overlord and k8s control plane. You can try out this feature using the following config:
druid.indexer.runner.useK8sSharedInformers=true
cgroup v2 support
cgroup v2 is now supported, and all cgroup metrics now emit cgroupversion to identify which version is being used.
The following metrics automatically switch to v2 if v2 is detected: CgroupCpuMonitor , CgroupCpuSetMonitor, CgroupDiskMonitor,MemoryMonitor. CpuAcctDeltaMonitor fails gracefully if v2 is detected.
Additionally, CgroupV2CpuMonitor now also emits cgroup/cpu/shares and cgroup/cpu/cores_quota.
Query reports for Dart
Dart now supports query reports for running and recently completed queries. The reports can be fetched from the /druid/v2/sql/queries/<sqlQueryId>/reports endpoint.
The format of the response is a JSON object with two keys, "query" and "report". The "query" key is the same info that is available from the existing /druid/v2/sql/queries endpoint. The "report" key is a report map including an MSQ report.
You can control the retention behavior for reports using the following configs:
druid.msq.dart.controller.maxRetainedReportCount: Max number of reports that are retained. The default is 0, meaning no reports are retaineddruid.msq.dart.controller.maxRetainedReportDuration: How long reports are retained in ISO 8601 duration format. The default isPT0S, meaning time-based expiration is turned off
New segment format
The new version 10 segment format improves upon version 9. It is off by default and not compatible with older segment format versions.
Set druid.indexer.task.buildV10=true to make segments in the new format.
If you downgrade, you must reindex your data with a supported segment format version.
You can use the bin/dump-segment tool to view segment metadata. The tool outputs serialized JSON.
Web console
New info available in the web console
The web console now includes information about the number of available processors and the total memory (in binary bytes).
This information is also available through the sys.servers table.
Other web console improvements
- Added tracking for inactive workers for MSQ execution stages #18768
- Added a refresh button for JSON views and stage viewers #18768
- You can now define
ARRAYtype parameters in the query view #18586 - Changed system table queries to now automatically use the native engine #18857
- Improved time charts to support multiple measures #18701
Ingestion
- Added support for AWS
InternalErrorcode retries #18720 - Improved ingestion to be more resilient. Ingestion tasks no longer fail if the task log upload fails with an exception #18748
- Improved how Druid handles situations where data doesn't match the expected type #18878
- Improved JSON ingestion so that Druid can compute JSON values directly from dictionary or index structures, allowing ingestion to skip persisting raw JSON data entirely. This reduces on-disk storage size #18589
- You can now choose between full dictionary-based indexing and nulls-only indexing for long/double fields in a nested column #18722
SQL-based ingestion
Additional ingestion configurations
You can now use the following configs to control how your data gets ingested and stored:
maxInputFilesPerWorker: Controls the maximum number of input files or segments per worker.maxPartitions: Controls the maximum number of output partitions for any single stage, which affects how many segments are generated during ingestion.
Other SQL-based ingestion improvements
- Added
maxRowsInMemoryto replacerowsInMemory.rowsInMemorynow functions as an alternate way to provide that config and is ignored ifmaxRowsInMemoryis specified. Previously, onlyrowsInMemoryexisted #18832
Streaming ingestion
Record offset and partition
You can now ingest the record offset (offsetColumnName) and partition (partitionColumnName) using the KafkaInputFormat. Their default names are kafka.offset and kafka.partition respectively .
Other streaming ingestion improvements
- Improved supervisors so that they can't kill tasks while the supervisor is stopping #18767
- Improved the lag-based autoscaler for streaming ingestion #18745
- Improved the
SeekableStreamsupervisor autoscaler to wait for tasks to complete before attempting subsequent scale operations. This helps prevent duplicate supervisor history entries #18715
Querying
Other querying improvements
- Improved the user experience for invalid
regex_expqueries. An error gets returned now #18762
Cluster management
Dynamic capacity for Kubernetes-based deployments
Druid can now dynamically tune the task runner capacity.
Include the capacity field in a POST API call to /druid/indexer/v1/k8s/taskrunner/executionconfig. Setting a value this way overrides druid.indexer.runner.capacity.
Server properties table
The sys.server_properties table exposes the runtime properties configured for each Druid server. Each row represents a single property key-value pair associated with a specific server.
Other cluster management improvements
- Added quality of service filtering for the Overlord so that health check threads don't get blocked #18033
Data management
Other data management improvements
- Added the
mostFragmentedFirstcompaction policy that prioritizes intervals with the most small uncompacted segments #18802 - Improved how segment files get deleted to prevent partial segment files from remaining in the event of a failure during a delete operation #18696
- Improved compaction so that it identifies multi-value dimensions for dimension schemas that can produce them #18760
- Improved the reliability when pulling segments #18821
Metrics and monitoring
Task metrics
All task metrics now emit the following dimensions: taskId, dataSource, taskType, groupId, and id. Note that id is emitted for backwards compatibility. It will be removed in favor of the taskId dimension in a future release.
Ingestion metrics
The following metrics for streaming and batch tasks now emit the actual values instead of 0: ingest/merge/time, ingest/merge/cpu, and ingest/persists/cpu.
statsd metrics
The following metrics have been added to the default list for statsd:
task/action/run/timetask/status/queue/counttask/status/updated/countingest/handoff/time
Other metrics and monitoring improvements
-
Added
reasondimension toingest/events/thrownAwaymetric. This allows for increased observability on why certain events are being logically excluded from ingest #188855 -
Added logging for all handlers for a stage before they start or stop, which can help you understand execution order #18662
-
Added new Jetty thread pool metrics to capture request-serving thread statistics:
jetty/threadPool/utilized,jetty/threadPool/readyandjetty/threadPool/utilizationRate#18883 -
Added
tierandprioritydimensions to thesegments/maxmetric #18890 -
Added
GroupByStatsMonitor, which includesdataSourceandtaskIddimensions for metrics emitted on peons #18711 -
Added
task/waiting/timemetric, which measures the time it takes for a task to be placed onto the task runner for scheduling and running #18735 -
Added the
supervisorIdto streaming task metrics to help clarify situations where multiple supervisors ingest data into a single datasource #18803 -
Added
StorageMonitortodruid.monitoring.monitorsto measure storage and virtual storage usage by segment cache #18742 -
Druid now logs the following:
-
Changed Prometheus emitter TTL tracking to consider all label value combinations instead of just the metric name. Labels aren't tracked when the TTL isn't set #18718 #18689
-
Changed lifecycle
stop()to be logged at theinfolevel to matchstart()#18640 -
Changed the trigger for metrics emission so that metrics get emitted any time a task completes #18766
-
Improved the metrics emitter so that it emits metrics for all task completions #18766
Extensions
SpectatorHistogram extension
- Added
SPECTATOR_COUNTandSPECTATOR_PERCENTILESQL functions #18885 - Improved the performance of the SpectatorHistogram extension through vectorization #18813
Upgrade notes and incompatible changes
Upgrade notes
MSQ controller tasks
When upgrading from Druid 30 or earlier, MSQ query_controller tasks can fail during a rolling update due to the addition of new counters that are not backwards compatible with these older versions. You can either retry any failed queries after the update completes; or you can set includeAllCounters to false in the query context for any MSQ jobs that need to run during the rolling update; or you can upgrade to Druid 31–35 first before upgrading to Druid 36.
Segment format
Druid 36.0.0 supports segment format version 10. Previous versions of Druid don't support version 10. If you downgrade, you must reindex your data with a supported segment format version.
Deprecated metrics
Monitors on peons that previously emitted the id dimension from JettyMonitor, OshiSysMonitor, JvmMonitor, JvmCpuMonitor, JvmThreadsMonitor and SysMonitor to represent the task ID are deprecated and will be removed in a future release. Use the taskId dimension instead.
Removed metrics
The following obsolete metrics have been removed:
segment/cost/raw#18846segment/cost/normalized#18846segment/cost/normalization#18846task/action/log/time#18649
Developer notes
Segment file interfaces
New SegmentFileBuilder and SegmentFileMapper interfaces have been defined to replace direct usages of FileSmoosher and SmooshedFileMapper to abstract the segment building and reading process.
The main developer visible changes for extension writers with custom column implementations is that the Serializer interface has changed the writeTo method:
- It now accepts a
SegmentFileBuilderinstead of aFileSmoosher - The
ColumnBuildermethodgetFileMappernow returns aSegmentFileMapperinstead ofSmooshedFileMapper.
Extensions which do not provide custom column implementations should not be impacted by these changes.
Other developer improvements
- Added the ability to override the default Kafka image for testing #18739
- Changed
fastDecompressortosafeDecompressor#18930 - Extensions can now provide query kit implementations #18875
- Removed version overrides in individual
pomfiles. For a full list, see the pull request #18708
Dependency updates
The following dependencies have had their versions bumped:
org.apache.logging.log4j:log4j-corefrom2.22.1to2.25.3#18874org.mozilla:rhinofrom1.7.14to1.7.14.1#18868net.java.dev.jnaandnet.java.dev.jnaversions from5.13.0to5.18.1for Oshi monitor #18848com.github.oshi:oshi-core fromfrom6.4.4to6.9.1#18839bcpkix-jdk18onfrom1.78.1to1.79#18834org.eclipse.jettyfrom12.0.25to12.0.30#18773hamcrestfrom1.3to2.2#18708org.apache.commons:commons-lang3from3.18.0to3.19.0#18695org.apache.maven.plugins:maven-shade-pluginfrom3.5.0to3.6.1com.netflix.spectatorfrom1.7.0to1.9.0#18887org.bouncycastle:bcpkix-jdk18onfrom1.79to1.81to resolveSONATYPE-2025-001911
Credits
@317brian
@abhishekrb19
@AdheipSingh
@aho135
@Akshat-Jain
@amaechler
@ashwintumma23
@capistrant
@cecemei
@clintropolis
@cryptoe
@dependabot[bot]
@EdwinIngJ
@Fly-Style
@GabrielCWT
@gargvishesh
@gianm
@GWphua
@hfukada
@inponomarev
@jtuglu1
@keoliva
@kfaraz
@kgyrtkirk
@Pankaj260100
@Rasnar
@rohangarg
@Shiyang-Zhao
@TessaIO
@tinnou
@uds5501
@vogievetsky
@vtlim
@writer-jill
Apache Druid 35.0.1 is a patch release that contains important fixes for historical server segment dropping and the protobuf input format extension.
For information about new features in Druid 35, see the Druid 35 release notes.
Bug fixes
- Fixed an issue where dropping segments didn't remove the memory mapping of segment files, leaving file descriptors of the deleted files open until the process exits #18782
- Fixed an issue where the URL path for the protobuf input format wasn't loading #18770
Dependency updates
- Updated lz4-java to 1.8.1 #18804
Apache Druid 35.0.0 contains over 229 new features, bug fixes, performance enhancements, documentation improvements, and additional test coverage from 29 contributors.
See the complete set of changes for additional details, including bug fixes.
Review the upgrade notes and incompatible changes before you upgrade to Druid 35.0.0.
If you are upgrading across multiple versions, see the Upgrade notes page, which lists upgrade notes for the most recent Druid versions.
# Important features, changes, and deprecations
This section contains important information about new and existing features.
# Jetty
Druid 35 uses Jetty 12. This change may impact your deployment. For more information,see the upgrade note for Jetty 12
# Java support
Druid now supports Java 21. Note that some versions of Java 21 encountered issues during test, specifically Java 21.05-21.07. If possible, avoid these versions.
Additionally, support for Java 11 has been removed. Upgrade to Java 17 or 21.
# Projections
Projections have been improved:
- Static filters are now supported
- The granularity in queries can match UTC time zones
Additionally, there have been general improvements to performance and reliability.
# Virtual storage (experimental)
Virtual storage fabric mode for enables Historical servers to serve more segments than what their physical disk can hold. Instead of loading segments when they're published, segments get loaded on demand during queries. Any unneeded segments get removed from disk but get loaded again when processing a query requires that particular segment.
To enable virtual storage, set druid.segmentCache.virtualStorage to true.
This feature is experimental and has several limitations. For more information, see the pull request:
# New input format
Druid now supports a lines input format. Druid reads each line from an input as UTF-8 text and creates a single column named line that contains the entire line as a string. Use this for reading line-oriented data in a simple form for later processing.
# Multi-stage query task engine
The MSQ task engine is now a core capability of Druid rather than an extension. It has been in the default extension load list for several releases.
Remove druid-multi-stage-query from druid.extensions.loadList in common.runtimes.properties before you upgrade.
Druid 35.0.0 will ignore the extension if it's in the load list. Future versions of Druid will fail to start since it can't locate the extension.
# Improved monitor loading
You can now specify monitors in common.runtime.properties and each monitor will be loaded only on the applicable server types. Previously, you needed to define monitors in the specific runtime.properties file for the service a monitor is meant for.
# Exact count extension
A new contributor extension (druid-exact-count-bitmap) adds support for exact cardinality counting using Roaring Bitmap over a Long column.
# Improved indexSpec
Users can now specify a format specification for each JSON column individually, which will override the IndexSpec defined in the ingestion job. Additionally, a system-wide default IndexSpec can be set using the druid.indexing.formats.indexSpec property.
# Functional area and related changes
This section contains detailed release notes separated by areas.
# Web console
# Time zones
You can now configure whether the web console displays local time or UTC. This setting is stored locally in your browser and doesn't impact other users.
Note that the URL maintains the query parameters in UTC time, but the Druid console automatically converts the filter to local time.
# Other web console improvements
- Added better support for MSQ task engine-based compaction tasks. They now use the stages pane to render the compaction report instead of showing the JSON #18545
- Added a version column to the Services tab so that you can see what version a service is running. This is helpful during rolling upgrades to verify the state of the cluster and upgrade #18542
- Improved the resiliency of the web console when the supervisor history is extensive #18416
# Ingestion
# Dimension schemas
At ingestion time, dimension schemas in dimensionsSpec are now strictly validated against allowed types. Previously an invalid type would fall back to string dimension. Now, such values are rejected. Users must specify a type that's one of the allowed types. Omitting type still defaults to string, preserving backward compatibility.
# Other ingestion improvements
- Added support for session tokens (
sessionToken) to the S3 input source #18609 - Improved performance of task APIs served by the Overlord. Druid now reads the in-memory state of the Overlord before fetching task information from the metadata store #18448
- Improved task execution so that they can successfully complete even if there are problems pushing logs and reports to deep storage #18210
# SQL-based ingestion
# Other SQL-based ingestion improvements
- Added the ability to configure the maximum frame size. Generally, you don't need to change this unless you have very large rows #18442
- Added logging for when segment processing fails #18378
- Improved logging to store the cause of invalid field exceptions #18517 #18517
# Streaming ingestion
# Other streaming ingestion improvements
- Added a count parameter to the supervisor history API #18416
# Querying
# Segment timeout
You can now configure timeout for a query's segment processing.
To set a timeout, set druid.processing.numTimeoutThreads to a low number, such as 3, on all data nodes where you want this feature enabled.
Then, include the perSegmentTimeout query context parameter in your query and set it to a value in milliseconds.
# Caching scan query results
Druid now supports result-level caching for scan queries.
By default, this behavior is turned off. To enable, override the druid.*.cache.unCacheable property.
# New expressions for sketches
Druid now supports the following expressions:
HLL_SKETCH_ESTIMATE_WITH_ERROR_BOUNDSTHETA_SKETCH_ESTIMATE_WITH_ERROR_BOUNDS
These estimates work on a sketch column and have the same behavior as the post aggregators.
# New multi-value SQL functions
Druid now supports the following multi-value functions:
MV_FILTER_REGEX: filters a multi-value expression to include only values matching the specified regular expression patternMV_FILTER_PREFIX: filters a multi-value expression to include only values that start with the specified prefix
# Exception handling
You can now write exceptions that Druid encounters as a row and then verify whether the exception is expected. To use this feature, set the writeExceptionBodyAsResponseRow query context parameter to true.
# Other querying improvements
- Added stricter validation for
GREATESTandLEAST#18562 - Added a query context option called
realtimeSegmentsOnlythat returns results from realtime segments only when set totrue#18329 - Improved the performance of the following query types through vectorization:
- Improved the deserialization time for intervals by more than 40% #18477
- Improved the performance of
AND/ORas well asNOT/IS TRUE/IS FALSE#18491 #18488 - Improved the performance of scan queries #18441
- Improved the performance of metadata queries by parallelizing them at the data-node level #18592
# Cluster management
# Labels for the metadata table
You can now configure labels for Druid services. Use the druid.labels field in the configuration options for the service and provide a JSON object of key-value pairs, such as druid.labels={"location":"Airtrunk"} or druid.labels.location=Airtrunk.
These labels can be queried using the sys.servers system table and are also displayed on the Services tab in the web console.
# TLS server configs
Add a druid.server.https.forceApplyConfig config flag to indicate that the TLSServerConfig values should be applied irrespective of whether a preexisting SslContextFactory.Server binding exists.
# Error message strategy
You can now configure Druid to log an error message and return an error ID for non-user targeted messages. This can be configured by setting druid.server.http.errorResponseTransform.strategy to persona.
# Kerberos authentication
The druid.auth.authenticator.kerberos.cookieSignatureSecret config is now mandatory.
# Other cluster management improvements
- Added a
versioncolumn tosys.server. This is useful during rolling upgrades to verify the state of the cluster #18542 - Added support for short, unique index names in metadata stores #18515
- Added config
stopTaskCountPercentto lag-based auto-scaler to allow task count to be scaled as a percentage of the current task count. #18480 - Added headers to the user agent for services, improving traceability between services #18505
- Added a new Kafka emitter config:
druid.emitter.kafka.producer.shutdownTimeout. It controls how long the Kafka producer waits for pending requests to finish before shutting down #18427 - Changed how Netty worker threads get calculated, lowering the number of threads used at small processor counts #18493
- Changed the response for the
/status/propertiesto be sorted lexicographically on the property names. The response is more readable now and related keys are grouped together #18506. - Improved segment loading during startup in certain situations by optimizing concurrent writing #18470
# Data management
- Improved S3 storage to support storing segments in S3 without zip compression #18544
# Metrics and monitoring
# Improved Prometheus emitter
You can now specify the TTL for a metric. If a metric value has not been updated within the TTL period then the value will stop being emitted.
Set flushPeriod when using the exporter strategy in the prometheus-emitter.
# Other metrics and monitoring improvements
- Added a
statusCodedimension toquery/timeandsqlQuery/timemetrics containing the query result status code #18633 #18631 - Added
segment/schemaCache/refreshSkipped/countmetric that shows the number of segments that skipped schema refresh #18561 - Added a metric for when auto-scaling /gets skipped when the publishing task set is not empty #18536
- Added Kafka consumer metrics for indexing tasks in the
prometheus-emitterdefault metrics config #18458 - Added an event when a datasource gets dropped from the Coordinator:
segment/schemaCache/datasource/dropped/count#18497 - Added the
taskTypedimension to the following Prometheus metrics to distinguish between realtime and compaction tasks:ingest/count,ingest/segments/count, andingest/tombstones/count#18452 - Added a server dimension to segment assignment skip metrics like
segment/assignSkipped/count#18313 - Changed
CgroupUtilto be less noisy #18472 - Reduced the clutter of output logs when there's a mismatch in Kafka and task partition set #18234
# Extensions
# Iceberg
You can now filter Iceberg data based on a configurable time window.
# Kubernetes
# HTTP client
You can now configure what HTTP client library the Fabric8 KubernetesClient uses under the hood to communicate with the k8s server tasks run on. The default remains the same client and config that Druid 34 uses. The additional options that are now available are okhttp and a native JDK HttpClient. The default client and config should suffice for most use cases.
Set the druid.indexer.runner.k8sAndWorker.http.httpClientType config to vertx (default), okhttp, or jdk.
# Pod logs
You can now set a timeout for asynchronous operations that interact with k8s pod logs: druid.indexer.runner.podLogOperationTimeout.
# Task logs
For Middle Manager-less ingestion, persisting task logs no longer have the potential to block indefinitely. Instead, there is a time limit for persisting logs, that if breached, results in giving up on the persist. The default timeout is 5 minutes, but can be configured by overriding druid.indexer.runner.logSaveTimeout with a duration, such as PT60S.
Additionally, task log storage for the Kubernetes runner is now optional.
# Improved fault tolerance
The druid-kubernetes-overlord-extensions that uses Kubernetes for launching and managing tasks instead of the Middle Managers is now more resilient against certain transient faults.
# Faster task failure
Task pods will now output Kubernetes job events when failing during the pod creation phase, and K8s tasks will fail fast under unrecoverable conditions.
# Mount additional configs
Previously, you could specify additional configuration to be mounted to _common via an environment variable, but this only applied to the Druid process. You can now also apply the same configuration to ingestion tasks submitted as Kubernetes jobs (MiddleManager-less mode), ensuring consistency across all task types.
# Upgrade notes and incompatible changes
# Upgrade notes
# Fallback vectorization on by default
The druid.expressions.allowVectorizeFallback now defaults to true. Additionally, SAFE_DIVIDE can now vectorize as a fallback.
# Jetty 12
Druid now uses Jetty 12. Your deployment may be impacted depending, specifically with regards to URI compliance and SNI host checks.
A new server configuration option has been added: druid.server.http.uriCompliance. Jetty 12 by default has strict enforcement of RFC3986 URI format. This is a change from Jetty 9. To avoid potential breaking changes in existing Druid deployments, this config defaults to LEGACY, which uses the more permissive URI format enforcement that Jetty 9 used. If the cluster you operate does not require legacy compatibility, we recommend you use the upstream Jetty default of RFC3986 in your Druid deployment. See the jetty documentation for more info.
Jetty 12 servers do strict SNI host0 checks when TLS is enabled. If the host your client is connecting to the server with does not match what is in the keystore, even if there is only one certificate in that keystore, it will return a 400 response. This could impact some use cases, such as folks connecting over localhost for whatever reason. If this change will break your deployment, you can opt-out of the change by setting druid.server.http.enforceStrictSNIHostChecking to false in the runtime.properties for some or all of your Druid services. It is recommended that you modify your client behavior to accommodate this change in jetty instead of overriding the config whenever possible.
# Kerberos authentication
The druid.auth.authenticator.kerberos.cookieSignatureSecret config is now mandatory.
# Multi-stage query task engine
The MSQ task engine is now a core capability of Druid rather than an extension. It has been in the default extension load list for several releases.
Remove druid-multi-stage-query from druid.extensions.loadList in common.runtimes.properties before you upgrade.
Druid 35.0.0 will ignore the extension if it's in the load list. Future versions of Druid will fail to start since it cannot locate the extension.
# pac4j extension
Due to the upgrade from pac4j 4 to 5, session serialization has changed from pac4j’s JavaSerializer to standard Java serialization. As a result, clients of clusters using the pac4j extension may be logged out during rolling upgrades and need to re‑authenticate.
# Incompatible changes
# Java 11 support removed
Upgrade to Java 17 or 21. Note that some versions of Java 21 encountered issues during test, specifically Java 21.05-21.07. If possible, avoid these versions.
# Developer notes
# Embedded cluster tests
You can use the new embedded cluster test framework to run lightweight integration tests using a fully functional Druid cluster that runs on a single process, including any extensions you want to load as well as ZooKeeper and Derby metadata store.
You no longer need to run mvn build, create a docker image, or even run startup scripts to perform integration tests.
# Specialized virtual columns for JSON
The ColumnHolder returned by ColumnSelector now returns a SelectableColumn rather than BaseColumn from getColumn(). The SelectableColumn may be a BaseColumn or may be an implementation provided by a virtual column. Extensions that need to retrieve physical columns from a ColumnSelector can check if the column is a BaseColumn using instanceof. For convenience, the return type of getColumnHolder on QueryableIndex has been specialized to BaseColumnHolder, which always returns BaseColumn from getColumn(). Extensions that are calling this method on QueryableIndex can store the result in a variable of type BaseColumnHolder to avoid needing to cast the column.
# reset-cluster tool removed
The reset-cluster dev and testing environment oriented tool has been removed. It was outdated and would fail to completely clear out the metadata store (missing upgradeSegments and segmentSchemas tables added in Druid 28 and 30 respectively). This tool was not intended for production use cases, so there should be no impact for cluster operators.
# Breaking change for SQL metadata extensions
The MetadataStorageActionHandler extension point now has simpler APIs.
This is a breaking change for any SQL metadata extensions in the wild.
The changes for PostgreSQL, MySQL and SQLServer are already complete. #18479
# DriudLeaderClient removed
DruidLeaderClient has been removed from extensions and replaced with a ServiceClient-based implementation. Additionally, CoordinatorServiceClient for querying non-core APIs exposed by the Coordinator is available through druid-server.
# Dependency updates
The following dependencies have been updated:
pac4jversion 4.5.7 to 5.7.3 #18259nimbus-jose-jwtversion 8.22.1 to 9.37.2 #18259oauth2-oidc-sdkversion 8.22 to 10.8 #18259avacticaversion 1.25.0 to 1.27.0 #18421 #18671protobuf-dynamiclibrary replaced with a built-in protobuf method #18401com.github.spotbugs:spotbugs-maven-pluginversion 4.8.6.6 to 4.9.3.2 #18177com.google.code.gsonversion 2.10.1 to 2.12.0 #18527org.apache.commonsversion 3.17.0 to 3.18.0 #18572fabric8version 7.2.0 to 7.4 #18556jacksonversion 2.18.4 to 2.19.2 #18556
# Credits
@317brian
@a2l007
@abhishekagarwal87
@abhishekrb19
@adarshsanjeev
@aho135
@Akshat-Jain
@amaechler
@ashibhardwaj
@ashwintumma23
@capistrant
@cecemei
@clintropolis
@cryptoe
@dependabot[bot]
@GabrielCWT
@gianm
@GWphua
@jamangstangs
@jtuglu1
@kfaraz
@kgyrtkirk
@maytasm
@rasmusselsmark
@skuldshao
@uds5501
@vogievetsky
@writer-jill
@yurmix
@ Every self-hosted release, weekly
One Monday email with the week's stable releases across the directory.
FAQ
Is Druid in-memory?
No longer strictly: Druid memory-maps data between disk and memory, so a node can load up to its disk size and clusters can be configured anywhere from all-in-memory to heavily over-committed, tiered by query requirements.
Where does Druid fit in my big data stack?
Druid typically sits behind a raw-data source such as a message bus like Apache Kafka or a filesystem like HDFS, ingests an indexed columnar copy, and serves analytics workloads to applications and users.
How is Druid deployed?
On commodity hardware in any *NIX environment; the cluster is made up of several services (ingestion, querying, coordination, etc.) that can be co-located on the same machines, and it runs well in AWS, GCP, Azure and other clouds.
Does Druid separate storage and compute?
Druid queries its own indexed copy of the raw data, called a segment, instead of pulling raw data from an external storage system at query time.
Is Druid a timeseries database? When should I use Druid over InfluxDB/OpenTSDB/Prometheus?
It shares traits with TSDBs and partitions data by time, but it combines analytics-database and search designs and can outperform TSDBs on non-time grouping, searching and complex metrics like histograms and quantiles.
Is Druid a log aggregation/log search system? When should I use Druid over Elastic/Splunk?
Not really: Druid uses compressed-bitmap inverted indexes for fast filtering, but it is an analytics engine rather than a text-log full-text search system; it is often used for semi-structured data such as JSON.
Is Druid a data warehouse? When should I use Druid over Redshift/BigQuery/Snowflake?
No: Druid is a new kind of database aimed at real-time analytic workloads over event-driven data, borrowing column-oriented storage from warehouses but also designs from search and time-series systems, with much lower query and ingest latency.
More in Data Warehousing
firebolt-coreA free, self-hosted analytical database with PostgreSQL SQL, Iceberg support, and distributed query execution scaling from laptop to enterprise clusters.
GizmoSQLSelf-hosted SQL server on DuckDB/SQLite with Arrow Flight SQL, TLS, JWT auth. Core is Apache 2.0; Enterprise adds SSO, metrics, permissions.
lakehouse-at-homeSelf-hosted lakehouse for local data engineering with Spark, Iceberg, and Kafka via Docker Compose, plus optional Airflow and test data tools.
PontoonPontoon is a self-hosted, open-source data export platform for vendors to send data directly to customer warehouses like Snowflake, BigQuery, Redshift, and Postgres.
clickhouseClickHouse is an open-source, self-hosted column-oriented DBMS for real-time OLAP analytics using SQL.
cloudberryOne advanced and mature open-source MPP (Massively Parallel Processing) database. Open source alternative to Greenplum Database.
d8aOpen-source analytics platform that collects GA4 and Matomo data into your own ClickHouse, BigQuery, or file storage with full privacy control.
databendSelf-hosted, open-source data warehouse in Rust for large-scale analytics plus vector and full-text search, with Python UDF sandboxes and git-like data branching.




0 comments