feat!: rewrite ruby api and frontend with stable-url autosourced feeds by gildesmarais · Pull Request #775 · html2rss/html2rss-web

gildesmarais · 2025-09-09T20:41:38Z

This pull request introduces a comprehensive overhaul of the development environment, CI workflows, and documentation to modernize the stack, improve developer experience, and streamline frontend/backend integration. The most important changes include the adoption of a unified Dev Container workflow with Docker Compose, a switch to Ruby 3.4.6 and Node.js 22 for both local and CI environments, a modernized CI pipeline with full frontend and backend test coverage, and updated documentation and coding standards for contributors.

Development Environment Modernization

Introduced a new Dev Container setup using Docker Compose, with a custom Dockerfile, docker-compose.yml, and updated devcontainer.json to provide an integrated Ruby (3.4.6) and Node.js (22.19.0) environment, exposing ports 4000 (Ruby app) and 4001 (Astro frontend). The workspace and gem cache are mounted for faster development. ([[1]](https://github.com/html2rss/html2rss-web/pull/775/files#diff-13bd9d7a30bf46656bc81f1ad5b908a627f9247be3f7d76df862b0578b534fc6R1-R34), [[2]](https://github.com/html2rss/html2rss-web/pull/775/files#diff-67a4805fdcc6145d7b3ada2a6099a9b2e91c9d0fd108c22f95d2f01d219793d1R1-R21), [[3]](https://github.com/html2rss/html2rss-web/pull/775/files#diff-24ad71c8613ddcf6fd23818cb3bb477a1fb6d83af4550b0bad43099813088686L3-R41), [[4]](https://github.com/html2rss/html2rss-web/pull/775/files#diff-751af1a340658c7b8176fe32d7db9fadbe15d1d075daba1919a91df04155bc70R1-R2))
Added detailed Dev Container workflow documentation in .devcontainer/README.md and AGENTS.md, including common commands, verification steps, and collaboration guidelines. ([[1]](https://github.com/html2rss/html2rss-web/pull/775/files#diff-0f9548cbce10dbdef0b86d783ed0727a4ac1061fc71cb549b5dbd400705c71fdR1-R49), [[2]](https://github.com/html2rss/html2rss-web/pull/775/files#diff-a54ff182c7e8acf56acfd6e4b9c3ff41e2c41a31c9b211b2deb9df75d9a478f9R1-R82))
Updated VS Code recommendations and settings for improved Ruby and Astro development, including formatting, linting, and file associations. ([[1]](https://github.com/html2rss/html2rss-web/pull/775/files#diff-c16655a98a3ee89a7636a59c59a72b0e93649e3a1e947327cfc43a1336b4e912R1-R7), [[2]](https://github.com/html2rss/html2rss-web/pull/775/files#diff-a5de3e5871ffcc383a2294845bd3df25d3eeff6c29ad46e3a396577c413bf357R1-R24))

Continuous Integration & Testing Improvements

Replaced the legacy CI workflow with a new .github/workflows/ci.yml that runs on push, PR, and manual triggers, with concurrency control. The pipeline now includes Ruby linting, tests, OpenAPI verification, frontend build and tests, Docker smoke tests, and conditional Docker publishing. ([[1]](https://github.com/html2rss/html2rss-web/pull/775/files#diff-b803fcb7f17ed9235f1e5cb1fcd2f5d3b2838429d4368ae4c57ce4436577f03fL1-R20), [[2]](https://github.com/html2rss/html2rss-web/pull/775/files#diff-b803fcb7f17ed9235f1e5cb1fcd2f5d3b2838429d4368ae4c57ce4436577f03fR29-L51), [[3]](https://github.com/html2rss/html2rss-web/pull/775/files#diff-b803fcb7f17ed9235f1e5cb1fcd2f5d3b2838429d4368ae4c57ce4436577f03fR143-R157))
Ensured that both Ruby and Node.js versions are sourced from .tool-versions for consistency across environments. ([.tool-versionsR1-R2](https://github.com/html2rss/html2rss-web/pull/775/files#diff-751af1a340658c7b8176fe32d7db9fadbe15d1d075daba1919a91df04155bc70R1-R2))
Removed the old bundle update workflow (bundle-update.yml) and legacy Ruby version file. ([[1]](https://github.com/html2rss/html2rss-web/pull/775/files#diff-d0a3f3ed056ec82f4ec49d4ff0019a13c2a83d312061273cdc585fc9f83d49ebL1-L17), [[2]](https://github.com/html2rss/html2rss-web/pull/775/files#diff-1bdb3279c7718f7037156f83c80eb81c58d37b286e027219e055b76249082264L1))

Frontend and Backend Integration

Updated the main Dockerfile to perform a multi-stage build: first building the Astro frontend, then bundling Ruby gems, and finally assembling the runtime image with the built frontend. The runtime now uses Ruby 3.4.6 and exposes port 4000 by default. ([[1]](https://github.com/html2rss/html2rss-web/pull/775/files#diff-dd2c0eb6ea5cfc6c4bd4eac30934e2d5746747af48fef6da689e85b752f39557L1-R11), [[2]](https://github.com/html2rss/html2rss-web/pull/775/files#diff-dd2c0eb6ea5cfc6c4bd4eac30934e2d5746747af48fef6da689e85b752f39557L25-R49))
Updated health check endpoints and environment variables to use token-based authentication and the new port. ([DockerfileL25-R49](https://github.com/html2rss/html2rss-web/pull/775/files#diff-dd2c0eb6ea5cfc6c4bd4eac30934e2d5746747af48fef6da689e85b752f39557L25-R49))

Documentation & Coding Standards

Expanded and clarified .github/copilot-instructions.md with updated project structure, rules, and explicit Do's and Don'ts for both frontend and backend development. Added verification steps and emphasized the use of Astro for the frontend. ([[1]](https://github.com/html2rss/html2rss-web/pull/775/files#diff-227c2c26cb2ee0ce0f46a320fc48fbcbdf21801a57f59161b1d0861e8aad55f5L6-R56), [[2]](https://github.com/html2rss/html2rss-web/pull/775/files#diff-227c2c26cb2ee0ce0f46a320fc48fbcbdf21801a57f59161b1d0861e8aad55f5R65-R70))
Improved RuboCop configuration for modern plugins, file exclusions, and disabled class documentation requirement. ([[1]](https://github.com/html2rss/html2rss-web/pull/775/files#diff-4f894049af3375c2bd4e608f546f8d4a0eed95464efcdea850993200db9fef5cL1-R1), [[2]](https://github.com/html2rss/html2rss-web/pull/775/files#diff-4f894049af3375c2bd4e608f546f8d4a0eed95464efcdea850993200db9fef5cR10-R18), [[3]](https://github.com/html2rss/html2rss-web/pull/775/files#diff-4f894049af3375c2bd4e608f546f8d4a0eed95464efcdea850993200db9fef5cR28))
Added .yardopts for Yard documentation, and Spectral/OpenAPI config files for API linting. ([[1]](https://github.com/html2rss/html2rss-web/pull/775/files#diff-e7a0e8e7ae667738a817654cb27a45913c478f1aeca9619fa938069f9193c17fR1-R10), [[2]](https://github.com/html2rss/html2rss-web/pull/775/files#diff-d5da7cb43c444434994b76f3b04aa6e702c09e938de09dbc09d72569d611d9abR1), [[3]](https://github.com/html2rss/html2rss-web/pull/775/files#diff-d021d4b6613f6566813a00e96546d6e7c3199d7b115d276e19a0ebd0aec0cf6fR1-R5))

Summary:
This PR modernizes the development workflow with a Docker-based Dev Container, aligns Ruby and Node.js versions across environments, overhauls CI for full-stack testing, and updates documentation and standards for a streamlined, collaborative developer experience.

References:
[1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21] [22] [23]

…rver script

… instructions

- Deleted test scripts for auto-source and URL restrictions as they are no longer needed. - Removed unused dependencies from package.json and package-lock.json to streamline the project. - Updated the dev script to improve clarity in error messages regarding port conflicts.

…ibility toggles

Upgrade docker smoke coverage to execute both feature modes in CI by passing explicit SMOKE_AUTO_SOURCE_ENABLED through workflow matrix and rake orchestration. Decision: keep the existing smoke entrypoint while making mode intent explicit to prevent blind spots and feature-toggle regressions. This is the highest-impact, low-complexity reliability gain for pre-merge confidence.

Extract feed endpoint orchestration into CreateFeed and ShowFeed command modules to reduce the monolithic Feeds service surface. Decision: preserve existing behavior and test seams by keeping Api::V1::Feeds as a thin delegator with extract_site_title compatibility passthrough. This makes endpoint logic easier to reason about and lowers regression risk for future feed changes.

…snapshot model Simplify AccountManager by deriving frozen account/token structures per call and dropping class-level mutable memoization, eliminating thread-safety snowflake patterns. Decision: keep reload! as a no-op compatibility hook and reset it in test setup so existing interfaces stay stable while behavior is deterministic. Also adjusted health failure spec isolation to avoid coupling to account-loading internals after cache removal.

…efaults Reintroduce immutable account snapshot memoization with explicit reload semantics to avoid per-request config parsing while keeping deterministic behavior. Normalize container smoke/runtime defaults to port 4000 and add a v2 migration guide documenting health routes, auth semantics, feature-flag policy, and deployment steps. This keeps the architecture consistent for a major-version transition with clear operator guidance.

Rewrite the migration guide for operators, integrators, and contributors with complete scope: architecture shifts, breaking behavior changes, workflow/CI changes, rollout steps, and rollback boundaries. This replaces the minimal delta notes with an audience-ready major-version migration document aligned to the actual branch rewrite size.

…specs Unify API success envelopes and health/feed routing behavior so runtime output and documented contract stay aligned. Adopt a Rails-free OpenAPI pipeline using rspec-openapi, generated from request specs, and enforce drift checks in CI via regenerate+diff. Also add cache lifecycle logging around account snapshots to make reload/build behavior observable during ops and test runs. Outlier scope: this combines contract generation, API response shape alignment, and observability improvements because they share a single rollback boundary around v1 API correctness and release-readiness.

Make OpenAPI maintenance explicit for contributors by adding first-class Make targets for generation and drift verification. Document the exact commands in README so local behavior matches CI and contract updates are intentional rather than ad-hoc. This commit includes the regenerated OpenAPI artifact to keep docs aligned with the current request-spec source of truth.

Expose the generated OpenAPI contract at /api/v1/openapi.yaml so clients and tooling can fetch a canonical machine-readable schema from the running service. Add openapi_url to /api/v1 metadata to make contract discovery explicit and reduce client-side path assumptions. Keep the endpoint resilient during OpenAPI regeneration by returning a minimal fallback document when the generated file is temporarily absent, avoiding self-referential failures in spec generation flows.

Use netstat-based port detection instead of lsof in bin/dev because BusyBox lsof returns false positives in the dev container. Also replace the fixed 3s Puma readiness check with a bounded wait loop so slower boots do not fail make dev prematurely.

Apply actionable review feedback by replacing hardcoded sample tokens with explicit CHANGE_ME placeholders across config/runtime/docs and aligning smoke/API test defaults. Also fix devcontainer Prettier config path, deny URL access when allowed_urls is empty, short-circuit API routing before static routes, and return string XML bodies from feed generation. Add UrlValidator specs for empty allowlist denial and explicit wildcard allowance.

Align guest demo, guest auth, and signed-in screens to a single operational layout so the product feels consistent across states and easier to scan. Prioritize one primary action per state, reduce visual competition, and keep supporting actions lightweight to improve first-run clarity and conversion flow. Keep this as one intentional cross-cutting UX outlier because hierarchy, copy tone, interaction rhythm, and component primitives needed to change together to avoid partial-language regressions. Follow-up: lock the one-off UX spec as acceptance criteria and extend E2E/visual checks to prevent future drift between states.

Reduce cognitive load in the core conversion flow by tightening vertical rhythm, minimizing secondary copy, and lowering non-essential visual emphasis. Preserve one clear primary task while keeping account/meta and advanced utilities available but unobtrusive. Includes targeted test updates to keep behavior and contract coverage aligned with the simplified UI structure.

Ensure the frontend dev server is reachable from the host by publishing container port 4001 in the devcontainer compose setup. Avoid Vite dep-scan failures during Astro dev by constraining optimization entries to app sources and excluding msw/node, which is test-only and not valid for browser dep optimization. This keeps the local dev loop reliable without changing runtime production behavior.

@param

Add concise class/module intent docs and method YARD tags across API v1 files. Include @param, @return, and @option return typing for stable payload contracts while keeping comments focused on boundary rationale (KISS).

Add a strict YARD verification task for public Ruby methods and wire it into make ready/lint-ruby. Expand essential YARD coverage across core app boundaries and document agent expectations for non-public methods when helpful.

Center the app on conversion as the primary action across guest, member, and result states by enforcing a single compact panel grammar and member-only advanced controls. Move bookmarklet rendering into the Preact member flow so guests only see demo onboarding, and keep authenticated context in one consistent panel-meta row. Redesign result output for utility-first scanning: fixed-width layout, title-first hierarchy, single copy CTA, subscribe action, and parsed feed-item preview with silent-fail behavior so URL actions remain reliable even when preview fetch/parsing fails. This is a deliberate cross-cutting UI outlier commit because the behavior depends on coordinated updates to panel structure, shared styling, and tests; rollback boundary is this commit.

Reduce misleading accent usage in the result view so non-interactive labels read as informational, not clickable. Align action grammar by left-aligning the reset CTA and tightening vertical spacing around result metadata for a denser, more consistent layout. Expose friendlier strategy display names from the API (Standard (recommended), JavaScript pages) so frontend selectors communicate intent instead of implementation details. Includes matching test/mock updates to preserve contract and UI behavior guarantees.

…tions Anchor architecture delivery to the current code reality so execution can proceed autonomously with low migration risk. The plan now explicitly validates assumptions against existing Roda routes, config/auth/feed boundaries, and current OpenAPI/frontend tooling before proposing changes. Sequence decisions prioritize reversible slices: typed config and observability before boundary contract migration, with async refresh intentionally last due to operational risk. Outlier handling is documented via commit packaging guidance so cross-cutting work remains explicit and rollback boundaries stay clear.

Introduce a browser-level smoke check so delivery can be self-verified beyond unit/contract tests. Given Alpine-based dev containers, Playwright now targets the container-native Chromium path for reliable headless execution instead of relying on incompatible downloaded browser binaries. Keep scope intentionally narrow: one smoke scenario that validates onboarding/auth mode transitions, with Vitest explicitly excluding e2e specs to avoid framework cross-collection. This provides an immediate autonomous verification path while preserving existing test semantics and pre-commit gates.

Establish explicit architectural decisions before implementation so autonomous delivery has clear guardrails and rollback boundaries. The ADR set locks migration policy for Data contracts, request-context observability, typed config snapshots, OpenAPI-generated frontend client workflow, and async feed refresh strategy. This keeps later implementation commits focused on execution rather than design churn.

…able feed runtime Execute the backend architecture phases by introducing immutable boundary contracts and typed config snapshots while keeping compatibility adapters for existing hash consumers. Request context is now initialized at middleware entry and propagated into security/error observability so auth, token usage, and failures correlate by request metadata. A cache-first feed runtime with feature-flagged async refresh is introduced behind safe defaults, with extraction into dedicated handler modules to keep the Roda app surface maintainable. Outlier note: this is intentionally cross-cutting because these runtime contracts interlock; rollback remains bounded via adapters and ASYNC_FEED_REFRESH_ENABLED=false.

Replace handwritten API wiring with generated OpenAPI client usage so frontend behavior tracks backend contract changes by construction. Add generation and verification commands plus Makefile integration to fail fast on spec/client drift instead of relying on manual sync. Hook and test updates preserve existing UX behavior while adapting error handling to generated client semantics.

Establishes phase-gated governance artifacts for AppContext, feature flags, observability events, and API contract policy. Locks a code-backed assumptions baseline in the delivery plan so implementation decisions stay grounded in current behavior. Decision: preserve doc-first phase gates while keeping lightweight pre-work inventory in the flags model.

Outlier: combines AppContext and Flags foundations because boot wiring in app.rb now depends on both to remain runnable and testable in each commit. Decision: make dependency resolution explicit via AppContext and remove direct feature-flag ENV reads from runtime paths in favor of a typed registry. Adds fail-fast flag validation boundaries and a focused flag-spec suite to keep startup behavior deterministic.

Adds explicit event emission for auth, feed creation, feed rendering, and error handling using the documented schema fields. Decision: keep existing security logs intact and layer observability events additively to avoid operational blind spots during transition. Failure events are emitted at boundary points where outcome is final to reduce duplicate noise.

Extends contract enforcement beyond spec generation drift to include OpenAPI linting and frontend generated-client drift verification. Decision: keep policy checks in CI so contract integrity is enforced consistently across contributors and environments.

- Drop surface card from guest panel — all panels now share `workspace` open layout - Replace radio cards with a plain <select> for demo source (consistent with Strategy selector) - Align button vocabulary: btn--ghost for demo run, btn--accent for primary auth/convert actions - "Back to demo" demoted to btn--meta (gray housekeeping, not invitation) - Remove ~95 lines of dead CSS: surface*, onboarding-*, radio-card/list*, fieldset/legend, form--spacious, auth-form-actions Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

- useStrategies: init isLoading as !!token to prevent empty-select flash on first render - Strategy select always rendered (disabled + Loading… placeholder while fetching), no layout jump - Improved hint copy: "Direct fetch — works for most sites. Fast and safe." / "Headless browser — use for JavaScript-rendered pages (React, Angular, SPAs)." Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

gildesmarais added the WIP work in progress label Sep 9, 2025

gildesmarais moved this to In progress in Project Board Sep 18, 2025

gildesmarais added this to Project Board Sep 18, 2025

gildesmarais force-pushed the feat/revamp-frontend branch 2 times, most recently from 3cd0291 to 26085bf Compare September 27, 2025 14:50

gildesmarais added 2 commits January 25, 2026 15:56

feat: astro frontend

f46ccfa

integrate astro

e7cf89f

gildesmarais force-pushed the feat/revamp-frontend branch from 26085bf to a58186e Compare January 25, 2026 15:04

gildesmarais added 22 commits January 25, 2026 16:23

auto reload

dbb68c5

feat: enhance development experience with live reload and new Ruby se…

2bf5773

…rver script

feat: enhance auto source functionality with authentication and usage…

92a065b

… instructions

autosource views in astro

edf4072

clean roda app

870327c

unify make lint & ~fix

cae7703

fix ctrl+c make dev

632b40f

make lintfix

47b982b

integrate astro-starlight

6ba5790

add api to return available request strategies

270bdb7

slim frontend, add ux-friendly autosource auth using hmac + docs

7162569

improve auto_source ux

d031291

style and ux

80d6726

refactor: roda app structure / routes

caa892d

tool-versions it is

3bfa3ac

ruby clean

85683fb

refactor: enhance routing and response handling; improve frontend vis…

08b3d4d

…ibility toggles

simplecov coverage

3dec638

csp + error

7f7a39e

rack attack

02ac635

log for rack-attack

df5600b

gildesmarais added 13 commits February 28, 2026 22:37

gildesmarais changed the title ~~feat!: astro frontend~~ feat(frontend): unify and simplify conversion UX Mar 1, 2026

gildesmarais and others added 14 commits March 1, 2026 10:56

docs(api): add essential YARD docs for v1 boundaries

8ac0680

Add concise class/module intent docs and method YARD tags across API v1 files. Include @param, @return, and @option return typing for stable payload contracts while keeping comments focused on boundary rationale (KISS).

docs(yard): enforce typed public method docs

c475120

Add a strict YARD verification task for public Ruby methods and wire it into make ready/lint-ruby. Expand essential YARD coverage across core app boundaries and document agent expectations for non-public methods when helpful.

gildesmarais changed the title ~~feat(frontend): unify and simplify conversion UX~~ feat!: rewrite ruby api and frontend with stable-url autosourced feeds Mar 1, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat!: rewrite ruby api and frontend with stable-url autosourced feeds#775

feat!: rewrite ruby api and frontend with stable-url autosourced feeds#775
gildesmarais wants to merge 91 commits intomasterfrom
feat/revamp-frontend

gildesmarais commented Sep 9, 2025 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

gildesmarais commented Sep 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

gildesmarais commented Sep 9, 2025 •

edited

Loading