Add external data processing and storage section by splindsay-92 · Pull Request #3033 · ably/docs

splindsay-92 · 2025-12-15T18:15:54Z

Description

Originally this was just about adding a guide on data processing, but it's since expanded into a rework of the existing data-storage use-case guide into an expanded feature set under the Chat document root directory.

We now have a seperate section that deals with external storage and processing, under which data extraction, processing and storage all sit under.

Checklist

Commits have been rebased.
Linting has been run against the changed file(s).
The PR adheres to the writing style guide and contribution guide.

coderabbitai · 2025-12-15T18:16:01Z

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment
Commit unit tests in branch chat/external-data-processing

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

AndyTWF · 2026-01-22T10:25:33Z

src/pages/docs/chat/external-storage-and-processing/data-extraction.mdx

@@ -0,0 +1,552 @@
+---
+title: "Extract messages via integrations"


We do already have https://ably.com/docs/guides/chat/export-chat - is this intended to replace that?

Yes, off the back of conversations with Mark, it seemed this was less a use-case guide and more a how-to feature. I'll remove the file now as the information is contained in the new pages :)

AndyTWF · 2026-01-22T10:27:45Z

src/pages/docs/chat/external-storage-and-processing/data-extraction.mdx

+```
+</Code>
+
+## Understanding channel names and room names <a id="channel-names"/>


I think there's benefit here of encouraging users to group their chat rooms under a common namespace for simplicity, e.g gamechat:<room>

AndyTWF · 2026-01-22T10:39:18Z

src/pages/docs/chat/external-storage-and-processing/data-extraction.mdx

+### Example webhook handler
+
+<Code>
+```javascript


Does this add any value to the narrative? It's very abstract and is fairly obvious

AndyTWF · 2026-01-22T10:39:38Z

src/pages/docs/chat/external-storage-and-processing/data-extraction.mdx

+```
+</Code>
+
+## Using queues <a id="queues"/>


Would bump below streaming

AndyTWF · 2026-01-22T10:49:18Z

src/pages/docs/chat/external-storage-and-processing/data-processing.mdx

+  await room.messages.send({
+    text: responseText
+  }, {
+    extras: {


This approach doesn't work with Chat - metadata goes into data and headers go specifically into extras.headers

(Something we should think about addressing)

Removed for now till we decide how to handle this

AndyTWF · 2026-01-22T10:51:13Z

src/pages/docs/chat/external-storage-and-processing/data-processing.mdx

+await room.messages.send({
+  text: "Hello!",
+  metadata: {
+    intent: 'translate',


Would this decision be a client-side one, or something that the server would decide? If it's client side then you're up for spoofing.

Also for translation specifically on-device translation is something we'd consider

I've switched the example to focus more on something like a notification for now

AndyTWF · 2026-01-22T11:02:54Z

src/pages/docs/chat/external-storage-and-processing/data-storage.mdx

+- **Consistency**: Webhook calls that fail will lead to inconsistencies between your data storage and Ably, which can be difficult to resolve. Detect if this happens using the `[meta]log` channel and use the [history endpoint](#history-endpoint) to backfill missing data.
+- **[At-least-once delivery](/docs/platform/architecture/idempotency#protocol-support-for-exactly-once-delivery)**: You need to handle duplicate messages. Deduplication can be done by checking `serial` and `version.serial`.
+
+## Using outbound streaming <a id="outbound-streaming"/>


Some of these sections feel like repeating the above?

m-hulbert

This is pretty much there - mostly minor comments throughout except for the data storage page that is a little difficult to follow.

src/pages/docs/chat/external-storage-and-processing/index.mdx

m-hulbert · 2026-02-10T09:43:48Z

src/pages/docs/chat/external-storage-and-processing/index.mdx

+
+Chat rooms are built on Ably Pub/Sub channels, allowing you to leverage the full range of Ably's [platform integrations](/docs/platform/integrations) to forward messages to external systems.
+
+## Common use cases <a id="use-cases"/>


It's not best practice to have two headers following one another without any text if we can add some please.

m-hulbert · 2026-02-10T09:43:59Z

src/pages/docs/chat/external-storage-and-processing/index.mdx

+
+### Process messages with external systems <a id="process-messages"/>
+
+Extract and process messages through external systems, then respond back to the chat in real-time:


realtime throughout please.

m-hulbert · 2026-02-10T09:44:34Z

src/pages/docs/chat/external-storage-and-processing/index.mdx

+
+Extract and process messages through external systems, then respond back to the chat in real-time:
+
+* **AI-powered assistance**: Process commands through language models and respond with helpful information


Full stops at the end of bullet points please.

m-hulbert · 2026-02-10T09:45:22Z

src/pages/docs/chat/external-storage-and-processing/index.mdx

+
+* **AI-powered assistance**: Process commands through language models and respond with helpful information
+* **Real-time translation**: Automatically translate messages for multilingual chat rooms
+* **Content moderation**: Filter messages through moderation services


Should we include moderation here given we have a dedicated section on it?

Removed in favour of sentiment analysis 8ce1744

m-hulbert · 2026-02-10T10:22:33Z

src/pages/docs/chat/external-storage-and-processing/data-processing.mdx

+
+Each decoded message provides the following data for your processing logic. See [decoding messages](/docs/chat/external-storage-and-processing/data-extraction#decoding) for how to decode the integration envelope.
+
+* `text` is the message content, typically user input. See [message structure](/docs/chat/rooms/messages#structure).


I think this would be easier to read in a table.

m-hulbert · 2026-02-10T10:23:56Z

src/pages/docs/chat/external-storage-and-processing/data-processing.mdx

+
+## Handle message reactions <a id="reactions"/>
+
+Reaction summaries arrive as messages with `action` set to `message.summary`. If your processing needs reaction data, for example analytics on popular reactions or triggering actions when a message reaches a reaction threshold, extract reaction counts from the annotations payload.


This is useful info about extracting reaction counts that we don't cover in the extraction page.

I've made it clear in both now I think, so a smaller mention here as a reminder.

m-hulbert · 2026-02-10T10:24:27Z

src/pages/docs/chat/external-storage-and-processing/data-storage.mdx

+meta_keywords: "chat storage, data storage, data retention, compliance, analytics, message history, chat export"
+---
+
+Store chat messages from Ably Chat in your own data storage for long-term retention, compliance, analytics, or to maintain your data storage as the canonical source of truth.


"your own data storage" or "your own data store"?

I'll go with data store

m-hulbert · 2026-02-10T10:29:18Z

src/pages/docs/chat/external-storage-and-processing/data-storage.mdx

+
+While Ably Chat provides flexible data retention for messages (30 days by default, up to a year on request), this page discusses options for longer-term storage and additional control over your chat data.
+
+## What Ably Chat history gives you <a id="history-endpoint"/>


This page is a lot of bullet points that asks a lot of questions that we don't answer or offer any suggestions for.

I've done a bit of an overhaul here, removing bullet points and at the same time trying to focus more on the typical way to do things. 085941c

m-hulbert · 2026-02-10T10:30:05Z

src/pages/docs/chat/external-storage-and-processing/data-storage.mdx

+
+This approach provides full control over the publishing flow with immediate consistency (no integration delay), the opportunity to add validation or business logic before publishing, and the ability to use the Chat REST API or SDK to publish.
+
+**Considerations:**


We've had this as a header in other pages - but I wonder if we should offer less text for the less recommended approaches?

I've made the change, see what you think and I can always add it back in if your think we are now lacking in that area :)

m-hulbert

Looks like a lot of comments but this is 99% just suggestions for removing unnecessary links and including them within text 🙂

src/pages/docs/chat/external-storage-and-processing/index.mdx

src/pages/docs/chat/external-storage-and-processing/data-extraction.mdx

src/pages/docs/chat/external-storage-and-processing/data-processing.mdx

src/pages/docs/chat/external-storage-and-processing/data-storage.mdx

m-hulbert · 2026-02-11T11:55:37Z

src/pages/docs/chat/external-storage-and-processing/data-storage.mdx

+
+#### Data store schema design <a id="schema-design"/>
+
+Design your schema to support the features you need while keeping scale and reliability in mind. An early decision is whether you need to store all versions of messages or just the latest version (see [Decision 1](#decision-1) below), and whether to store full message objects or normalized relational data.


I'd actually suggest removing the "decision" terminology from here and the headings below and just make them open questions.

m-hulbert

👌

…g guide

…n and rename the page.

… section

…action guide.

Add redirect_from for old export-chat guide, reword intro, remove moderation bullet, and guide links into section intros.

> Change integration methods to types. > move channel names section into filtering > various other ordering and structural changes. > Add code examples for order handling.

> convert message data bullet list to a table

splindsay-92 force-pushed the chat/external-data-processing branch 4 times, most recently from 81034ca to 141f0a2 Compare December 18, 2025 21:26

m-hulbert added the review-app Create a Heroku review app label Dec 19, 2025

ably-ci temporarily deployed to ably-docs-chat-external-czhado December 19, 2025 11:43 Inactive

splindsay-92 requested a review from m-hulbert January 5, 2026 11:01

splindsay-92 force-pushed the chat/external-data-processing branch from a491a26 to 6b0ffcc Compare January 5, 2026 18:38

splindsay-92 requested a review from AndyTWF January 13, 2026 13:37

splindsay-92 force-pushed the chat/external-data-processing branch from 6b0ffcc to 2be4b41 Compare January 21, 2026 13:40

splindsay-92 marked this pull request as ready for review January 21, 2026 13:40

splindsay-92 changed the title ~~Add external data processing guide~~ Add external data processing and storage section Jan 22, 2026

AndyTWF added review-app Create a Heroku review app and removed review-app Create a Heroku review app labels Jan 22, 2026

ably-ci temporarily deployed to ably-docs-chat-external-vfcvkb January 22, 2026 10:20 Inactive

AndyTWF reviewed Jan 22, 2026

View reviewed changes

splindsay-92 force-pushed the chat/external-data-processing branch from 2be4b41 to dcfe862 Compare February 2, 2026 18:58

m-hulbert temporarily deployed to ably-docs-chat-external-ievfum February 9, 2026 16:48 Inactive

m-hulbert reviewed Feb 10, 2026

View reviewed changes

splindsay-92 requested a review from m-hulbert February 10, 2026 16:44

m-hulbert temporarily deployed to ably-docs-chat-external-ievfum February 11, 2026 09:46 Inactive

m-hulbert reviewed Feb 11, 2026

View reviewed changes

splindsay-92 requested a review from m-hulbert February 11, 2026 14:29

m-hulbert temporarily deployed to ably-docs-chat-external-ievfum February 11, 2026 14:52 Inactive

m-hulbert approved these changes Feb 11, 2026

View reviewed changes

splindsay-92 requested a review from AndyTWF February 16, 2026 14:47

AndyTWF approved these changes Feb 19, 2026

View reviewed changes

splindsay-92 added 2 commits February 20, 2026 10:05

docs: add guide for integrating external systems with Ably Chat

dc907a3

wip

f32de31

splindsay-92 added 27 commits February 20, 2026 10:05

Rework language and sentiment analysis examples in the data-processin…

4a69cf9

…g guide

Removing old integrations guide

7209896

Reduce use of bullet point sections

b2c6cd5

Rename "Message storage and history" to "Message history"

d15402d

Reduce bullet point sections and rework language to data store.

93d0fa3

Update and rework "Data storage" page to more the chat history sectio…

bb9bba7

…n and rename the page.

Remove message extraction guide and navigation link from main feature…

45ce54d

… section

wip

c75c262

wip

7eb6322

Reorder some sections and update "streaming" content in the data extr…

dce3606

…action guide.

wip

4785e86

wip

cc3a02f

Remove mention of old guide

8b01670

Clean up sections in data extraction.

a97d3e6

Rework data-processing to focus on message payloads and structure.

65ac88f

Metadata usage should be used over headers.

f19e255

wip

c540fe2

update namespace sections

8933047

Update metadata to include replies and mention notifications.

53836ac

Address index.mdx review comments.

ff1e659

Add redirect_from for old export-chat guide, reword intro, remove moderation bullet, and guide links into section intros.

Address review feedback on data extraction page

e5f4d26

> Change integration methods to types. > move channel names section into filtering > various other ordering and structural changes. > Add code examples for order handling.

Address review feedback on data processing page.

7c1d143

> convert message data bullet list to a table

Address review feedback on data processing page.

8b44a12

Add missing verison data to payload

d50e788

wip

93be4ac

wip

6b93c39

fixing up PR comments

f63e940

splindsay-92 force-pushed the chat/external-data-processing branch from 987997f to f63e940 Compare February 20, 2026 10:05

splindsay-92 merged commit 24038b1 into main Feb 20, 2026
7 checks passed

splindsay-92 deleted the chat/external-data-processing branch February 20, 2026 12:46

		@@ -0,0 +1,552 @@
		---
		title: "Extract messages via integrations"


		Chat rooms are built on Ably Pub/Sub channels, allowing you to leverage the full range of Ably's [platform integrations](/docs/platform/integrations) to forward messages to external systems.

		## Common use cases <a id="use-cases"/>


		### Process messages with external systems <a id="process-messages"/>

		Extract and process messages through external systems, then respond back to the chat in real-time:


		Extract and process messages through external systems, then respond back to the chat in real-time:

		* AI-powered assistance: Process commands through language models and respond with helpful information


		Each decoded message provides the following data for your processing logic. See [decoding messages](/docs/chat/external-storage-and-processing/data-extraction#decoding) for how to decode the integration envelope.

		* `text` is the message content, typically user input. See [message structure](/docs/chat/rooms/messages#structure).


		## Handle message reactions <a id="reactions"/>

		Reaction summaries arrive as messages with `action` set to `message.summary`. If your processing needs reaction data, for example analytics on popular reactions or triggering actions when a message reaches a reaction threshold, extract reaction counts from the annotations payload.


		While Ably Chat provides flexible data retention for messages (30 days by default, up to a year on request), this page discusses options for longer-term storage and additional control over your chat data.

		## What Ably Chat history gives you <a id="history-endpoint"/>


		This approach provides full control over the publishing flow with immediate consistency (no integration delay), the opportunity to add validation or business logic before publishing, and the ability to use the Chat REST API or SDK to publish.

		Considerations:


		#### Data store schema design <a id="schema-design"/>

		Design your schema to support the features you need while keeping scale and reliability in mind. An early decision is whether you need to store all versions of messages or just the latest version (see [Decision 1](#decision-1) below), and whether to store full message objects or normalized relational data.

Comments

Conversation

splindsay-92 commented Dec 15, 2025 • edited by atlassian bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Uh oh!

coderabbitai bot commented Dec 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

m-hulbert left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

m-hulbert left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

splindsay-92 commented Dec 15, 2025 •

edited by atlassian bot

Loading

coderabbitai bot commented Dec 15, 2025 •

edited

Loading