Appearance
Datatargets
A datatarget represents an external system that Custobar should deliver data to, either by actively pushing it using HTTP calls (webhooks), or by arranging the data to be fetched from Custobar API.
There are two types of datatargets: Datasync and message datatargets. Datasync targets are used to synchronize customer, product, and other data in Custobar with external systems, for example, keeping a web shop up-to-date with customer data changes. Message datatargets act as message queues for messages destined to external systems, for example, sending push messages to a system handing them over to mobile apps.
Getting a list of all installed datatargets:
GET /api/datatargets/Creating a new datatarget:
POST /api/datatargets/ {...}Example: creating a new message datatarget
POST /api/datatargets/ {
"datatarget_type": "messages",
"name": "App Messages"
}json
{
"datatarget": {
"id": "4egvzlql7g5j",
"datatarget_type": "messages",
"name": "App Messages",
"description": "",
"created_at": "2019-09-12T12:15:04Z",
"enabled": true,
"outlets": [],
"last_message_number": 0
}
}Note that the new datatarget is assigned a unique identifier ("4egvzlql7g5j"), and it is enabled by default. The last_message_number is the serial number of the last message posted in this datatarget, starting from 1, so 0 means that there have been no messages posted to this datatarget yet. The datatarget id is used in the api calls to refer the particular datatarget.
Accessing a single datatarget:
GET /api/datatargets/{datatarget_id}/Retrieves the current state of a single datatarget. Example:
GET /api/datatargets/4egvzlql7g5j/json
{
"datatarget": {
"id": "4egvzlql7g5j",
"datatarget_type": "messages",
"name": "App Messages",
"description": "",
"created_at": "2019-09-12T12:15:04Z",
"enabled": true,
"outlets": [],
"last_message_number": 0
}
}Updating the datatarget:
PATCH /api/datatargets/{datatarget_id}/ {...}Example:
PATCH /api/datatargets/4egvzlql7g5j/ {
"name": "Production App Messages",
"description": "Messages going straight to the app users"
}json
{
"datatarget": {
"id": "4egvzlql7g5j",
"datatarget_type": "messages",
"name": "Production App Messages",
"description": "Messages going straight to the app users",
"created_at": "2019-09-12T12:15:04Z",
"enabled": true,
"outlets": [],
"last_message_number": 0
}
}Message datatargets
A message datatarget represents a queue of messages from Custobar to an external system. Messages can be posted manually to the datatarget using the API:
POST /api/datatargets/{datatarget_id}/post/ {
"messages": [...]
}In production setting, you rarely need to post message directly to a datatarget, but it is often useful for testing a new datatarget when setting it up. The messages property is a list of new messages, each one being a JSON object that is the message to be delivered. The structure of the messages is entirely free, and depends on the targeted system. Example:
POST /api/datatargets/4egvzlql7g5j/post/ {
"messages": [
{
"app_user_id": "1104912",
"title": "We have new content for you!",
"text": "Please open the app now."
},
{
"app_user_id": "1466991",
"title": "We have new content for you!",
"text": "Please open the app now, or later."
}
]
}json
{
"first_message_number": 1,
"messages_count": 2
}The return value tells that the message number assigned to the first posted message 1, and that 2 messages were succesfully posted.
Retrieving data from datatarget:
GET /api/datatargets/4egvzlql7g5j/retrieve/
```json
{
"last_message_number": 2,
"messages": [
{
"app_user_id": "1104912",
"title": "We have new content for you!",
"text": "Please open the app now."
},
{
"app_user_id": "1466991",
"title": "We have new content for you!",
"text": "Please open the app now, or later."
}
]
}The returned result contains a list of the retrieved messages, in the order they were posted, and last_message_number property, telling the serial number of the last message retieved in this api call. This number can be used as after parameter to the next call, to retrieve the messages in batches:
GET /api/datatargets/4egvzlql7g5j/retrieve/?after=2json
{
"messages": []
}There are no more messages after message 2, so the returned messages list is empty. In this case, there is no last_message_number, as there are no messages.
The retrieval api accepts optional limit parameter that limits the number of messages retrieved in one call. If not given, the limit defaults to 100. Example:
GET /api/datatargets/4egvzlql7g5j/retrieve/?limit=1json
{
"last_message_number": 1,
"messages": [
{
"app_user_id": "1104912",
"title": "We have new content for you!",
"text": "Please open the app now."
}
]
}Retrieval api makes it convenient to set up passive pull style data integration, where the other system periodically polls Custobar for new data. To set up a push style data integration, where Custobar actively posts updates to another system's API, an outlet is needed.
Webhook outlets
A webhook outlet posts new data appearing in a datatarget to an http api endpoint. It contains the necessary configuration to make the http request, and keeps track of which messages in the queue have been successfully delivered, retrying, and throttling the sending as necessary.
To set up a new webhook outlet to an existing datatarget:
POST /api/datatargets/{datatarget_id}/outlets/ {
"outlet_type": "webhook",
"request": {
"url": "https://api.example.com/app-messages/",
"method": "POST",
"headers": {
"Authorization": "Bearer ANVDOSNNONO4411NIUNIU"
},
"format": "json",
"content": {
"app_messages": "{(data)}"
},
"is_batched": true
}
}json
{
"outlet": {
"id": "4xxqua6cai76",
"outlet_type": "webhook",
"enabled": true,
"request": {
"url": "https://api.example.com/app-messages/",
"headers": {
"Authorization": "Bearer ANVDOSNNONO4411NIUNIU"
},
"content": {
"app_messages": "{(data)}"
},
"method": "POST",
"format": "json"
},
"is_batched": true,
"max_batch_size": 100,
"last_delivered_message_number": 0,
"delivered_batch_count": 0,
"min_request_interval": "0"
}
}Note that the request's configuration defines format as json. This results in the content of the request to be json encoded, and appropriate content-type header to be set ("Content-Type: application/json"). The "{(data)}" value in content is a special placeholder for the actual messages. In this case, the outlet is configured to send messages in batches ("is_batched": true), so this placeholder will be replaced with a list of messages in actual requests.
At this point, the outlet is enabled, but is has not have had time to deliver any messages yet, so last_delivered_message_number and delivered_batch_count are both 0. After a while, we can check the state again:
GET /api/datatargets/4egvzlql7g5j/outlets/4xxqua6cai76/json
{
"outlet": {
"id": "4xxqua6cai76",
"outlet_type": "webhook",
"enabled": true,
"request": {
"url": "https://api.example.com/app-messages/",
"headers": {
"Authorization": "Bearer ANVDOSNNONO4411NIUNIU"
},
"content": {
"app_messages": "{(data)}"
},
"method": "POST",
"format": "json"
},
"is_batched": true,
"max_batch_size": 100,
"last_delivered_message_number": 2,
"delivered_batch_count": 1,
"min_request_interval": "0"
}
}Now, last_delivered_message_number is 2, and delivered_batch_count is 1, indicating that all the two messages posted in the datatarget have been successfully delivered to the outlet endpoint.
Webhook failure recovery
If a request to the remote server fails, The outlet will try sending the data again after a while.
Outlet logs
Each webhook outlet keeps its own request log that can be retrieved to observe the flow of data through that outlet. The log is accessed with url:
GET /api/datatargets/{datatarget_id}/outlets/{outlet_id}/log/For example:
GET /api/datatargets/4egvzlql7g5j/outlets/4xxqua6cai76/log/json
{
"order": "descending",
"limit": 100,
"entries": [
{
"entry_number": 2,
"date": "2019-09-24T13:23:15.171326Z",
"batch_number": 1,
"batch_size": 2,
"first_message_number": 1,
"status": "OK",
"http_status": 200,
"request_time_ms": 2,
"http_request_available": true
},
{
"entry_number": 1,
"date": "2019-09-24T13:23:13.121333Z",
"batch_number": 1,
"batch_size": 2,
"first_message_number": 1,
"status": "FAIL",
"http_status": 503,
"request_time_ms": 12,
"http_request_available": true
}
]
}From the log, you can see that there are two entries, returned in descending order (most recent first). The last (least recent) entry has status "FAIL", meaning that the request to the remote server was not a success. It has http_status 503, Service Unavailable, meaning that the remote server was not available at the time of the request. The first (most recent) entry tells, however, that the request was retried two seconds later, and this time it was successful, with status "OK", and http_status 200.
Log entry properties
The possible properties of a log entry are:
- entry_number: Sequence number of the log entry, starting from 1.
- date: Timestamp of the log, in UTC timezone, with microsecond resolution. This is taken just before the request to the remote server was initiated.
- batch_number: Sequence number of the delivered batch, starting from 1.
- batch_size: Number of messages in this batch
- first_message_number: Sequence number of the first message is this batch (only in message datatargets)
- after_update: Identifier for the last update that was delivered before this batch (only in datasync datatargets)
- status: either "OK" or "FAIL", indicating if the request was successfully delivered
- http_status: The HTTP status code returned by the remote server
- fail_reason: A string detailing the reason for failure, if there was a failure, and there are details available
- http_request_available: Indicates if the actual HTTP request for this log entry is available for retrieval.
Details for a particular entry can be retrieved with api call:
GET /api/datatargets/{datatarget_id}/outlets/{outlet_id}/log/{entry_number}/Example:
GET /api/datatargets/4egvzlql7g5j/outlets/4xxqua6cai76/log/1/json
{
"entry": {
"entry_number": 1,
"date": "2019-09-24T13:23:13.121333Z",
"batch_number": 1,
"batch_size": 2,
"first_message_number": 1,
"status": "ERROR",
"http_status": 503,
"request_time_ms": 12,
"http_request": "POST /app-messages/ HTTP1.1\r\n...",
"http_reply": "'HTTP/1.1 503 ..."
}
}When retrieved individually, the entry includes the actual http request and reply, in the text form they were sent over the wire. The reply may not be present if there was no reply, for example if the request timed out before getting a reply.
Log pagination
The number of log entries retrieved in one request can be set using the limit url parameter
GET /api/datatargets/4egvzlql7g5j/outlets/4xxqua6cai76/log/?limit=1json
{
"order": "descending",
"limit": 1,
"total_count": 4,
"next_url": "https://COMPANY.custobar.com/api/datatargets/4egvzlql7g5j/outlets/4xxqua6cai76/log/?from=1&limit=1",
"entries": [
{
"entry_number": 2,
"entry_key": "p1c0",
"date": "2019-09-24T13:23:15.171326Z",
"batch_number": 1,
"batch_size": 2,
"first_message_number": 1
"status": "OK",
"http_status": 200,
"request_time_ms": 2,
"http_request_available": true
}
]The log entries are paged. If there are more than limit entries available, only the first limit log entries are returned in the reply. Next limit entries can be retrieved by calling the next_url included in the reply.
The reply includes total_count of all entries in the log. A custom slice of the log can be retrieved using from, limit, and order query parameters. For example, to retrieve 10 first log entries, except the very first one, in ascending order, call:
GET /api/datatargets/4egvzlql7g5j/outlets/4xxqua6cai76/log/?limit=10&from=2&order=ascendingOrder may be either ascending or descending (default, if order is not explicitly given).