Publishing Events

EndPoints

Raccoon can be hosted behind a proxy/API GW, a sample of blocks as below.

The HTTP API path which accepts events is:

/api/v1/events

This path can be used for sending events by either connecting via websocket connection or as a normal REST API Request.

HTTP methods used for Endpoints are:

• Websocket - GET
• REST - POST

Authorization/Authentication

Raccoon does not provide features to perform any authorization or authentication of the user/client initiating the connection. It accepts connections as trusted (and assumes any such auth is already performed)

HTTP backend

SSL termination is outside the scope of Raccoon, and the service API accepts HTTP connections assuming that the SSL is terminated at a proxy or ELB before reaching Raccoon.

gRPC backend

Similar to HTTP SSL termination is outside the scope of Raccoon, and the service API accepts connections assuming SSL is terminated at a proxy or ELB before reaching Raccoon.

Data Formatters

Raccoon supports Protos and JSON as the primary data formatters. Protobufs can be used to send event via websockets, REST or gRPC whereas JSON is supported only for websocket and REST endpoint.

With a websocket connection the content type is identified based on the mesage type. If the message type is binary it is assumed that the formatting is protobufs and if the message type is text then formatting is assumed to be JSON.

Protos

Raccoon accepts an SendEventRequest proto that wraps multiple Event proto. This enables clients to send an event in real-time or multiple events in batches.

Refer to raccoon.proto for how you can build the request.

message SendEventRequest {
  string req_guid = 1;
  google.protobuf.Timestamp sent_time = 2;
  repeated Event events = 3;
}

Where req_guid - A globally unique ID generated by the client denoting this request as unique sent_time - This is when the event is sent over the WebSocket in protobuf timestamp format. This time is used to calculate latencies in Raccoon. events - are events of type Event proto which is of the format

Refer to the Event.proto for how the event could be composed of.

message Event {
  // Data/byteArray of the serialised product proto.
  bytes eventBytes = 1;
  /* This is the protoMessageName which the protoc provides with each compiled proto. This type is used by raccoon to distribute events to respective Kafka topics.
  */
  string type = 2;
 }

Where eventBytes - is a byte array serialized by the event proto (eg. ViewedEvent.proto) serializer client type - event type which Raccoon uses to distribute the events to Kafka topics. More details in the following sections.

Clients build the event array and compose the SendEventRequest proto, send them through the WebSocket client.

Raccoon also wires response every time a message is read and processed.

Refer to the raccoon.proto that Raccoon sends for every event.

message SendEventResponse {
  Status status = 1;
  Code code = 2;
      /* time when the response is generated */
  int64 sent_time = 3;
      /* failure reasons if any */
  string reason = 4;
      /* Usually detailing the success/failures */
  map<string, string> data = 5;
}

enum Status {
    UNKNOWN_STATUS = 0;
      /* signifies request success */
    SUCCESS = 1;
      /* server request failures */
    ERROR = 2;
}

enum Code {
    UNKNOWN_CODE = 0;
          /* successfully read and deserialized */
    OK = 1;
          /* usually deserialization failures */
    BAD_REQUEST = 2;
          /* server runtime errors */
    INTERNAL_ERROR = 3;
          /* signifies max connection reached at the server */
    MAX_CONNECTION_LIMIT_REACHED = 4;
          /* signifies a user reached max connections allowed.Defaults to 1 */
    MAX_USER_LIMIT_REACHED = 5;
}

The above response model is self-explanatory. Clients can choose to retry for error codes such as Code=[3|4]

JSON

Sample JSON SendEventRequest

{
  "req_guid": "1234abcd",
  "sent_time": {
    "seconds": 1638154927,
    "nanos": 376499000
  },
  "events": [
    {
      "eventBytes": "Cg4KCHNlcnZpY2UxEgJBMRACIAEyiQEKJDczZTU3ZDlhLTAzMjQtNDI3Yy1hYTc5LWE4MzJjMWZkY2U5ZiISCcix9QzhsChAEekGEi1cMlNAKgwKAmlkEgJpZBjazUsyFwoDaU9zEgQxMi4zGgVBcHBsZSIDaTEwOiYKJDczZTU3ZDlhLTAzMjQtNDI3Yy1hYTc5LWE4MzJjMWZkY2U5Zg==",
      "type": "booking"
    }
  ]
}

Parameter	Data Type	Description
req_guid	string	A globally unique Identifier generated by the client.
sent_time	Object	Unix time in seconds+nanoseconds when the event is sent to Raccoon
events	Array(Object)	Array of event objects
events.eventBytes	string	base64 string of bytes generated by json serializion of EventProto
events.type	string	Event type which Raccoon uses to distribute the events to Kafka topics

Sample JSON SendEventResponse

{
  "status": 1,
  "code": 1,
  "sent_time": 1638155915,
  "data": {
    "req_guid": "1234abcd"
  }
}

Parameter	Data Type	Description
status	int	status of the send event request
code	int	response code
sent_time	int	sent time in seconds
data	int	data map sent by the server, currently contains just the req_guid
reason	string	reason for any failure if any

Values of status and codes is same as defined in Protos.

Headers

Raccoon service accepts headers to identify a user connection uniquely. The header name is made configurable as it enables clients to specify a header name that works for them. For, e.g. for a mobile app having a request header as X-User-ID which identifies the user (client) connecting to Raccoon, can configure Raccoon service with the config set as below SERVER_WEBSOCKET_CONN_ID_HEADER=X-User-ID. Optionally, SERVER_WEBSOCKET_CONN_GROUP_HEADER can also be configured to support multitenancy such as multiple apps connecting to a single Raccoon instance.

Raccoon uses the config to fetch the header name and uses the value passed in the request header with this name, as the connection id. This header name uniquely identifies a client. A client, in this case, can be the user in the app.

The following header is a sample providing a user id: 654785432. Once the client initiates a WebSocket upgrade request over Raccoon, assuming the request is upgraded, and the client connection is established, Racoon accepts the header and extracts the user id to build a connection map. This map helps deduplicate connections for a user within the same raccoon instance.

{
    "X-User-ID": "654785432"
}

Content-Type header is mandatory for sending event using REST API.

Following are the supported Content-Type headers for various data formats:

• Protobufs - application/proto
• JSON - application/json

gRPC

Events can be sent to Raccoon using gRPC too.

Refer to EventService.proto for the definition of EventService which exposes one RPC call SendEvent. It is recommended to generate the language specific gRPC client using the proto definition.

Input to the RPC call is SendEventRequest and the output is raccoon.proto.

To support multi-tenacy while using gRPC, SERVER_WEBSOCKET_CONN_ID_HEADER and SERVER_WEBSOCKET_CONN_GROUP_HEADER values can be used. The key along with their values if set in grpc metadata while sending the request. Golang client example -

md := metadata.New(map[string]string{"X-User-ID": "1234"})
ctx := metadata.NewOutgoingContext(context.Background(), md)
r, err := client.SendEvent(ctx, req)

Topics

Raccoon distributes events to a topic based on the event type. The protobufs section above clarifies how the type should be set in the event. The type is a string literal. For example, ViewedEvent - which signifies that the user viewed something on the app or the site can have its event type set as below type = viewedevent

When raccoon API consumes a batch array of events (events in SendEventRequest proto), it deserializes them and fetches the individual events (using the SendEventRequest proto), and constructs the topic to send each event to based on the type field set in each of the events.

The following code determines the topic name.

topic := fmt.Sprintf(pr.topicFormat, event.Type)

where topicformat - is the configured pattern EVENT_DISTRIBUTION_PUBLISHER_PATTERN type - is the type set by the client when the event proto is generated

For e.g. setting the

EVENT_DISTRIBUTION_PUBLISHER_PATTERN=topic-%s-log

and a type such as type=viewedevent in the event

will have the topic name as topic-viewedevent-log