Skip to content

Clean up failed clones and improve error logging#2188

Open
myieye wants to merge 1 commit intodevelopfrom
clean-up-failed-clones-and-improve-error-logging
Open

Clean up failed clones and improve error logging#2188
myieye wants to merge 1 commit intodevelopfrom
clean-up-failed-clones-and-improve-error-logging

Conversation

@myieye
Copy link
Collaborator

@myieye myieye commented Mar 3, 2026

We recently had a sync failure with the message:

Could not find file '/var/lib/fw-headless/projects/-5449db7a-fe73-4eb9-96fa-6f2f7641c506/fw/fw.fwdata'.

It took me a little while to realize how that happened: It was a WeSay project tagged as a FieldWorks project.
So, I decided:

  1. It should be more obvious to future devs what the likely cause is => I added a new error message for this case
  2. We should probably clean up failed clones as well as invalid projects like this one

That's what this PR does ☝️

@github-actions github-actions bot added 💻 FW Lite issues related to the fw lite application, not miniLcm or crdt related 📦 Lexbox issues related to any server side code, fw-headless included labels Mar 3, 2026
@coderabbitai
Copy link

coderabbitai bot commented Mar 3, 2026

📝 Walkthrough

Walkthrough

This change introduces a new custom exception (InvalidFwDataProjectException) to handle cases where FW data projects fail validation, expands error handling in the sync service to detect and respond to missing project files after cloning, adds a new sync status enum value (ProjectIncompatible), and updates test infrastructure to support scenarios where the FW data file is created at different points in the sync workflow.

Changes

Cohort / File(s) Summary
New Exception Class
backend/FwHeadless/Services/InvalidFwDataProjectException.cs
Introduces a sealed exception inheriting from InvalidOperationException with a ProjectFilePath property to capture the problematic project file path.
Sync Service Enhancement
backend/FwHeadless/Services/SyncHostedService.cs
Adds catch handler for InvalidFwDataProjectException in ExecuteSync; refactors SetupFwData to validate FW data file existence after clone, throw the new exception if missing, and clean up the project directory on clone failure.
Sync Status Enum Updates
backend/LexCore/Sync/SyncJobResult.cs, frontend/viewer/src/lib/dotnet-types/generated-types/LexCore/Sync/SyncJobStatusEnum.ts
Adds ProjectIncompatible enum member to SyncJobStatusEnum in both backend and frontend to represent sync jobs that encounter incompatible project structures.
Test Infrastructure
backend/Testing/FwHeadless/Services/SyncWorkerTestHarness.cs
Refactors RunAsync parameters from single createFwDataFile to separate createFwDataFileBeforeSync and createFwDataFileAfterClone; adds SetCloneResult method and EnsureFwDataFileExists helper to support testing various file creation scenarios.
Sync Tests
backend/Testing/FwHeadless/Services/SyncWorkerTests.cs
Adds two new test methods: ExecuteSync_CloneFailure_DeletesProjectFolder to verify cleanup on clone failure, and ExecuteSync_FwDataFileStillMissingAfterPreSetup_ReturnsUnableToSync to verify behavior when FW data file remains missing after pre-setup.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

Suggested reviewers

  • rmunn
  • hahn-kev

Poem

🐰 A project once failed to appear,
So we crafted exceptions sincere,
With cleanup so neat when clones go awry,
And statuses new help us know why—
InvalidFwData we now hold dear! 🌟

🚥 Pre-merge checks | ✅ 3
✅ Passed checks (3 passed)
Check name Status Explanation
Title check ✅ Passed The title 'Clean up failed clones and improve error logging' accurately summarizes the main changes: adding cleanup for failed clones and new error handling/logging for invalid projects.
Description check ✅ Passed The description clearly relates to the changeset by explaining the root cause (WeSay project mis-tagged as FieldWorks) and the two main improvements (better error messages and cleanup of failed clones).
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
  • 📝 Generate docstrings (stacked PR)
  • 📝 Generate docstrings (commit on current branch)
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch clean-up-failed-clones-and-improve-error-logging

Tip

Try Coding Plans. Let us write the prompt for your AI agent so you can ship faster (with fewer bugs).
Share your feedback on Discord.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions
Copy link
Contributor

github-actions bot commented Mar 3, 2026

UI unit Tests

  1 files  ±0   50 suites  ±0   25s ⏱️ -1s
138 tests ±0  138 ✅ ±0  0 💤 ±0  0 ❌ ±0 
203 runs  ±0  203 ✅ ±0  0 💤 ±0  0 ❌ ±0 

Results for commit aa7a4b8. ± Comparison against base commit 2b9fd3b.

♻️ This comment has been updated with latest results.

@myieye myieye requested a review from rmunn March 3, 2026 15:07
@github-actions
Copy link
Contributor

github-actions bot commented Mar 3, 2026

C# Unit Tests

164 tests   164 ✅  18s ⏱️
 23 suites    0 💤
  1 files      0 ❌

Results for commit aa7a4b8.

@argos-ci
Copy link

argos-ci bot commented Mar 3, 2026

The latest updates on your projects. Learn more about Argos notifications ↗︎

Build Status Details Updated (UTC)
default (Inspect) ✅ No changes detected - Mar 3, 2026, 3:09 PM

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (2)
backend/Testing/FwHeadless/Services/SyncWorkerTestHarness.cs (1)

115-121: Keep SetCloneResult behavior consistent with the default clone mock.

This helper currently skips the post-clone fwdata-file creation path that SetupDefaultMocks() applies. Aligning them reduces test setup surprises.

♻️ Proposed refactor
 public void SetCloneResult(SendReceiveHelpers.LfMergeBridgeResult result)
 {
     SendReceiveMock
         .Setup(s => s.Clone(It.IsAny<FwDataProject>(), ProjectCode))
-        .Callback(() => Steps.Add(Clone))
+        .Callback(() =>
+        {
+            Steps.Add(Clone);
+            if (result.Success && _createFwDataFileAfterClone) EnsureFwDataFileExists();
+        })
         .ReturnsAsync(result);
 }
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@backend/Testing/FwHeadless/Services/SyncWorkerTestHarness.cs` around lines
115 - 121, SetCloneResult currently stubs SendReceiveMock.Clone but omits the
post-clone fwdata-file creation that SetupDefaultMocks applies; update
SetCloneResult so its SendReceiveMock.Setup(s => s.Clone(...)) uses the same
Callback and post-clone behavior used in SetupDefaultMocks (i.e., call
Steps.Add(Clone) and invoke the same fwdata-file creation routine or helper that
SetupDefaultMocks triggers after clone) so tests using SetCloneResult mirror the
default clone mock behavior.
backend/Testing/FwHeadless/Services/SyncWorkerTests.cs (1)

306-307: Rename this test to match the asserted status.

The method name says ReturnsUnableToSync, but the assertion expects ProjectIncompatible. Renaming will make failures easier to interpret.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@backend/Testing/FwHeadless/Services/SyncWorkerTests.cs` around lines 306 -
307, The test method
ExecuteSync_FwDataFileStillMissingAfterPreSetup_ReturnsUnableToSync has a
mismatched name vs its assertion; rename the method to reflect the asserted
status (e.g.,
ExecuteSync_FwDataFileStillMissingAfterPreSetup_ReturnsProjectIncompatible) so
the test name matches the expected ProjectIncompatible outcome and update any
references to this method accordingly.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@backend/FwHeadless/Services/SyncHostedService.cs`:
- Around line 325-332: The current catch block that attempts to delete
fwDataProject.ProjectsPath can throw and replace the original clone/validation
exception; wrap the cleanup (Directory.Exists, logger.LogInformation,
Directory.Delete) in its own try-catch so any exception thrown during cleanup is
caught and logged (including the cleanup exception details and the path via
fwDataProject.ProjectsPath) but not rethrown, then rethrow the original
exception outside that inner try-catch to preserve the root cause; reference the
existing symbols fwDataProject.ProjectsPath, logger.LogInformation,
Directory.Delete and the surrounding catch/throw logic in SyncHostedService.cs.

---

Nitpick comments:
In `@backend/Testing/FwHeadless/Services/SyncWorkerTestHarness.cs`:
- Around line 115-121: SetCloneResult currently stubs SendReceiveMock.Clone but
omits the post-clone fwdata-file creation that SetupDefaultMocks applies; update
SetCloneResult so its SendReceiveMock.Setup(s => s.Clone(...)) uses the same
Callback and post-clone behavior used in SetupDefaultMocks (i.e., call
Steps.Add(Clone) and invoke the same fwdata-file creation routine or helper that
SetupDefaultMocks triggers after clone) so tests using SetCloneResult mirror the
default clone mock behavior.

In `@backend/Testing/FwHeadless/Services/SyncWorkerTests.cs`:
- Around line 306-307: The test method
ExecuteSync_FwDataFileStillMissingAfterPreSetup_ReturnsUnableToSync has a
mismatched name vs its assertion; rename the method to reflect the asserted
status (e.g.,
ExecuteSync_FwDataFileStillMissingAfterPreSetup_ReturnsProjectIncompatible) so
the test name matches the expected ProjectIncompatible outcome and update any
references to this method accordingly.

ℹ️ Review info

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 2b9fd3b and aa7a4b8.

📒 Files selected for processing (6)
  • backend/FwHeadless/Services/InvalidFwDataProjectException.cs
  • backend/FwHeadless/Services/SyncHostedService.cs
  • backend/LexCore/Sync/SyncJobResult.cs
  • backend/Testing/FwHeadless/Services/SyncWorkerTestHarness.cs
  • backend/Testing/FwHeadless/Services/SyncWorkerTests.cs
  • frontend/viewer/src/lib/dotnet-types/generated-types/LexCore/Sync/SyncJobStatusEnum.ts

Comment on lines +325 to +332
catch
{
logger.LogInformation("Clone result before CRDT sync: {Output}", srResult.Output);
if (Directory.Exists(fwDataProject.ProjectsPath))
{
logger.LogInformation("Cleaning up project folder after failed clone: {ProjectFolder}", fwDataProject.ProjectsPath);
Directory.Delete(fwDataProject.ProjectsPath, true);
}
throw;
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Don’t let cleanup failures mask the real clone error.

Directory.Delete(...) inside this catch can throw and replace the original clone/validation exception, which obscures root cause and complicates status reporting.

🛠️ Proposed fix
-            catch
+            catch
             {
-                if (Directory.Exists(fwDataProject.ProjectsPath))
+                try
                 {
-                    logger.LogInformation("Cleaning up project folder after failed clone: {ProjectFolder}", fwDataProject.ProjectsPath);
-                    Directory.Delete(fwDataProject.ProjectsPath, true);
+                    if (Directory.Exists(fwDataProject.ProjectsPath))
+                    {
+                        logger.LogInformation("Cleaning up project folder after failed clone: {ProjectFolder}", fwDataProject.ProjectsPath);
+                        Directory.Delete(fwDataProject.ProjectsPath, true);
+                    }
                 }
+                catch (Exception cleanupEx)
+                {
+                    logger.LogWarning(cleanupEx, "Failed to clean up project folder after failed clone: {ProjectFolder}", fwDataProject.ProjectsPath);
+                }
                 throw;
             }
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@backend/FwHeadless/Services/SyncHostedService.cs` around lines 325 - 332, The
current catch block that attempts to delete fwDataProject.ProjectsPath can throw
and replace the original clone/validation exception; wrap the cleanup
(Directory.Exists, logger.LogInformation, Directory.Delete) in its own try-catch
so any exception thrown during cleanup is caught and logged (including the
cleanup exception details and the path via fwDataProject.ProjectsPath) but not
rethrown, then rethrow the original exception outside that inner try-catch to
preserve the root cause; reference the existing symbols
fwDataProject.ProjectsPath, logger.LogInformation, Directory.Delete and the
surrounding catch/throw logic in SyncHostedService.cs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

💻 FW Lite issues related to the fw lite application, not miniLcm or crdt related 📦 Lexbox issues related to any server side code, fw-headless included

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant