Skip to main content

Introduction

Firehose is a cloud-native service for delivering real-time streaming data to destinations such as service endpoints (HTTP or GRPC) & managed databases (MongoDB, Prometheus, Postgres, InfluxDB, Redis, & ElasticSearch). With Firehose, you don't need to write applications or manage resources. It automatically scales to match the throughput of your data and requires no ongoing administration. If your data is present in Kafka, Firehose delivers it to the destination(SINK) that you specified.

Key Features​

Discover why users choose Firehose as their main Kafka Consumer

  • Sinks Firehose supports sinking stream data to log console, HTTP, GRPC, PostgresDB(JDBC), InfluxDB, Elastic Search, Redis, Prometheus and MongoDB.
  • Scale Firehose scales in an instant, both vertically and horizontally, for high-performance streaming sink and zero data drops.
  • Extensibility Add your own sink to Firehose with a clearly defined interface or choose from already provided ones.
  • Runtime Firehose can run inside containers or VMs in a fully managed runtime environment like Kubernetes.
  • Metrics Always know what’s going on with your deployment with built-in monitoring of throughput, response times, errors, and more.

Supported Incoming data types from kafka​

  • Protobuf
  • JSON
    • Supported limited to bigquery, elastic and mongo sink. In future support to other sinks will be added

Supported Sinks:​

Following sinks are supported in the Firehose

  • Log - Standard Output
  • HTTP - HTTP services (deprecated in v0.10.0+)
  • HTTP V2 - HTTP services
  • JDBC - Postgres DB
  • InfluxDB - A time-series database
  • Redis - An in-memory Key value store
  • ElasticSearch - A search database
  • GRPC - GRPC based services
  • Prometheus - A time-series database
  • MongoDB - A NoSQL database
  • Bigquery - A data warehouse provided by Google Cloud
  • Bigtable - A fully managed, scalable NoSQL database service for large analytical and operational workloads.
  • Blob Storage - A data storage architecture for large stores of unstructured data like google cloud storage, amazon s3, apache hadoop distributed filesystem

How can I get started?​

Explore the following resources to get started with Firehose:

  • Guides provide guidance on creating Firehose with different sinks.
  • Concepts describe all important Firehose concepts.
  • FAQs lists down some common frequently asked questions about Firehose and related components.
  • Reference contains details about configurations, metrics, FAQs, and other aspects of Firehose.
  • Contributing contains resources for anyone who wants to contribute to Firehose.