Skip to main content

HTTP

Note: HTTP sink type is deprecated from Firehose version 0.10.0 onwards. Please consider using HTTPV2 sink type instead.

REST API stands for Representational State Transfer and is an architectural pattern for creating web services.

An Http sink Firehose (SINK_TYPE=http) requires the following variables to be set along with Generic ones.

SINK_HTTP_SERVICE_URL

The HTTP endpoint of the service to which this consumer should PUT/POST/PATCH/DELETE data. This can be configured as per the requirement, a constant or a dynamic one (which extract given field values from each message and use that as the endpoint) If service url is constant, messages will be sent as batches while in case of dynamic one each message will be sent as a separate request (Since they’d be having different endpoints).

  • Example value: http://http-service.test.io
  • Example value: http://http-service.test.io/test-field/%%s,6 This will take the value with index 6 from proto and create the endpoint as per the template
  • Type: required

SINK_HTTP_REQUEST_METHOD

Defines the HTTP verb supported by the endpoint, Supports PUT, POST, PATCH and DELETE verbs as of now.

  • Example value: post
  • Type: required
  • Default value: put

SINK_HTTP_REQUEST_TIMEOUT_MS

Defines the connection timeout for the request in millis.

  • Example value: 10000
  • Type: required
  • Default value: 10000

SINK_HTTP_MAX_CONNECTIONS

Defines the maximum number of HTTP connections.

  • Example value: 10
  • Type: required
  • Default value: 10

SINK_HTTP_RETRY_STATUS_CODE_RANGES

Defines the range of HTTP status codes for which retry will be attempted. Please remove 404 from retry code range in case of HTTP DELETE otherwise it might try to retry to delete already deleted resources.

  • Example value: 400-600
  • Type: optional
  • Default value: 400-600

SINK_HTTP_DATA_FORMAT

If set to proto, the log message will be sent as Protobuf byte strings. Otherwise, the log message will be deserialized into readable JSON strings.

  • Example value: JSON
  • Type: required
  • Default value: proto

SINK_HTTP_HEADERS

Deifnes the HTTP headers required to push the data to the above URL.

  • Example value: Authorization:auth_token, Accept:text/plain
  • Type: optional

SINK_HTTP_JSON_BODY_TEMPLATE

Deifnes a template for creating a custom request body from the fields of a protobuf message. This should be a valid JSON itself.

  • Example value: {"test":"$.routes[0]", "$.order_number" : "xxx"}
  • Type: optional

SINK_HTTP_PARAMETER_SOURCE

Defines the source from which the fields should be parsed. This field should be present in order to use this feature.

  • Example value: Key
  • Example value: Message
  • Type: optional
  • Default value: None

SINK_HTTP_PARAMETER_PLACEMENT

Deifnes the fields parsed can be passed in query parameters or in headers.

  • Example value: Header
  • Example value: Query
  • Type: optional

SINK_HTTP_PARAMETER_SCHEMA_PROTO_CLASS

Defines the fully qualified name of the proto class which is to be used for parametrised http sink.

  • Example value: com.tests.TestMessage
  • Type: optional

INPUT_SCHEMA_PROTO_TO_COLUMN_MAPPING

Defines the mapping of the proto fields to header/query fields in JSON format.

  • Example value: {"1":"order_number","2":"event_timestamp","3":"driver_id"}
  • Type: optional

SINK_HTTP_OAUTH2_ENABLE

Enable/Disable OAuth2 support for HTTP sink.

  • Example value: true
  • Type: optional
  • Default value: false

SINK_HTTP_OAUTH2_ACCESS_TOKEN_URL

Defines the OAuth2 Token Endpoint.

  • Example value: https://sample-oauth.my-api.com/oauth2/token
  • Type: optional

SINK_HTTP_OAUTH2_CLIENT_NAME

Defines the OAuth2 identifier issued to the client.

  • Example value: client-name
  • Type: optional

SINK_HTTP_OAUTH2_CLIENT_SECRET

Defines the OAuth2 secret issued for the client.

  • Example value: client-secret
  • Type: optional

SINK_HTTP_OAUTH2_SCOPE

Space-delimited scope overrides. If scope override is not provided, no scopes will be granted to the token.

  • Example value: User:read, sys:info
  • Type: optional

SINK_HTTP_DELETE_BODY_ENABLE

This config if set to true will allow body for the HTTP DELETE method, otherwise no payload will be sent with DELETE request.

  • Example value: false
  • Type: optional
  • Default value: true

SINK_HTTP_SIMPLE_DATE_FORMAT_ENABLE

This config if set to true will enable the simple date format (Eg. - May 3, 2023 11:59:37 AM ) for timestamps (both at root and nested level also) in case of JSON payload. If set to false, it will send the timestamps (both at root and nested level also) in the ISO format (Eg. - 2023-05-03T11:59:36.965Z) . Note: This config is only applicable when the SINK_HTTP_JSON_BODY_TEMPLATE config is empty or not supplied.

  • Example value: false
  • Type: optional
  • Default value: true

SINK_HTTP_SERIALIZER_JSON_TYPECAST

Defines the mapping for field typecasting of the resulting JSON Serialization. This configuration could accept multiple mappings for multiple JSON Path Currently supported typecasting target: DOUBLE, INTEGER, LONG, STRING

  • Example value: [{"jsonPath": "$.root.someIntegerField", "type": "INTEGER"}, {"jsonPath": "$..[*].doubleField", "type": "DOUBLE"}]
  • Type: optional
  • Default Value: []