AIT-238 - handle publish failures

[JoaoDiasAbly]

Description

Context: https://ably.atlassian.net/browse/AIT-238

• when publishing or appending without waiting for an ack from the previous operation (pipelining) we can have one of the intermediary publishes/appends fail, creating a gap in the stream
• when such thing happens, we should "patch" the stream by:
• • message-per-response: publish the complete message with a different event type which should replace all tokens published before for that response
• • • some services already do this, so it's 1:1 mapping
• • • others don't, and the agent would need to buffers the response tokens on its side and patch the stream with the concatenation in case the publishing fails
• • message-per-token: patch the message with an edit operation containing the complete response (same concerns as above regarding how to implement and which services already do something similar)

Overall the questions I think need to be answers in these docs sections are:

• What happens if a publish/append fails?
• How to detect failures when you're not awaiting?
• How to handle/retry failed operations?
• What error scenarios to expect (rate limits, size limits, connection issues)?

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch AIT-238-handle-publish-failures

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[paddybyers]

Suggested clarification

[paddybyers]

Suggested change

	When appending without awaiting, it is possible for an intermediate append to fail while subsequent appends succeed. This creates a gap in the streamed response. For example, if a rate limit is exceeded, a single append may be rejected while the following tokens continue to be accepted.
	The examples above append successive tokens to a response message by pipelining the append operations - that is, the agent will publish an append operation without waiting for prior operations to complete. This is necessary in order to avoid the append rate being capped by the round-trip time from the agent to the Ably endpoint. However, this means that the agent does not await the outcome of each append operation, and that can result in the agent continuing to submit append operations after an earlier operation has failed. For example, if a rate limit is exceeded, a single append may be rejected while the following tokens continue to be accepted.
	The agent needs to obtain the outcome of each append operation, and take corrective action in the event that any operation failed for some reason. A simple but effective way to do this is simply to ensure that, if streaming of a response fails for any reason, then the message is updated with the final complete response text once it is available. This means that although the streaming experience is disrupted in the case of failure, there is no consistency problem with the final result once the response completes.

[paddybyers]

Suggested change

	To detect failures, keep a reference to each append operation and check for rejections after the stream completes:
	To detect append failures, keep a reference to each append operation and check for rejections after the stream completes:

[paddybyers]

I suggest updating this text in a similar way to proposed above

[GregHolmes]

Should we also be adding the other languages on the page for these code snippets? Currently your changes only support JavaScript, but we also have Java and Python. If these aren't relevant for this new section. We should add an If around the entire block of text and code so it only shows for JS.

I've also added a few comments. Thank you.

[GregHolmes]

Are we sure we want 2 Asides immediately after eachother? Feels like it breaks the flow of the page.

[JoaoDiasAbly]

addressed

[JoaoDiasAbly]

Should we also be adding the other languages on the page for these code snippets? Currently your changes only support JavaScript, but we also have Java and Python. If these aren't relevant for this new section. We should add an If around the entire block of text and code so it only shows for JS.

If we are adding Python and Java to the other AIT examples then I think it's also relevant here. I'm not confident to write to on either, so maybe that would need someone else to do it. Or I can try to have Claude have a go at it, at which point I might be able to review simple snippets and check for consistency with other examples. What do you think we should do here @GregHolmes ?

[GregHolmes]

Should we also be adding the other languages on the page for these code snippets? Currently your changes only support JavaScript, but we also have Java and Python. If these aren't relevant for this new section. We should add an If around the entire block of text and code so it only shows for JS.

If we are adding Python and Java to the other AIT examples then I think it's also relevant here. I'm not confident to write to on either, so maybe that would need someone else to do it. Or I can try to have Claude have a go at it, at which point I might be able to review simple snippets and check for consistency with other examples. What do you think we should do here @GregHolmes ?

If you get Claude to add them, then can always ask @ttypic to review those code snippets.

[JoaoDiasAbly]

Should we also be adding the other languages on the page for these code snippets? Currently your changes only support JavaScript, but we also have Java and Python. If these aren't relevant for this new section. We should add an If around the entire block of text and code so it only shows for JS.

If we are adding Python and Java to the other AIT examples then I think it's also relevant here. I'm not confident to write to on either, so maybe that would need someone else to do it. Or I can try to have Claude have a go at it, at which point I might be able to review simple snippets and check for consistency with other examples. What do you think we should do here @GregHolmes ?

If you get Claude to add them, then can always ask @ttypic to review those code snippets.

Pushed 165014d with the Python and Java examples.

[paddybyers]

lgtm

[JoaoDiasAbly]

@GregHolmes since @paddybyers has approved it, do you think we can merge it? Or would you rather still have someone validate the Java and Python examples?