Conversation
84a560c to
bd23487
Compare
04e5cc2 to
1446d54
Compare
...ive-executorch/common/rnexecutorch/models/instance_segmentation/BaseInstanceSegmentation.cpp
Outdated
Show resolved
Hide resolved
...ive-executorch/common/rnexecutorch/models/instance_segmentation/BaseInstanceSegmentation.cpp
Outdated
Show resolved
Hide resolved
...ive-executorch/common/rnexecutorch/models/instance_segmentation/BaseInstanceSegmentation.cpp
Outdated
Show resolved
Hide resolved
This comment was marked as resolved.
This comment was marked as resolved.
packages/react-native-executorch/src/modules/computer_vision/InstanceSegmentationModule.ts
Outdated
Show resolved
Hide resolved
...ative-executorch/common/rnexecutorch/models/instance_segmentation/BaseInstanceSegmentation.h
Outdated
Show resolved
Hide resolved
...ive-executorch/common/rnexecutorch/models/instance_segmentation/BaseInstanceSegmentation.cpp
Outdated
Show resolved
Hide resolved
packages/react-native-executorch/src/hooks/computer_vision/useInstanceSegmentation.ts
Outdated
Show resolved
Hide resolved
packages/react-native-executorch/common/rnexecutorch/models/instance_segmentation/Types.h
Outdated
Show resolved
Hide resolved
...ive-executorch/common/rnexecutorch/models/instance_segmentation/BaseInstanceSegmentation.cpp
Outdated
Show resolved
Hide resolved
...s/react-native-executorch/common/rnexecutorch/tests/integration/InstanceSegmentationTest.cpp
Show resolved
Hide resolved
packages/react-native-executorch/common/rnexecutorch/models/instance_segmentation/Types.h
Outdated
Show resolved
Hide resolved
packages/react-native-executorch/src/modules/computer_vision/InstanceSegmentationModule.ts
Show resolved
Hide resolved
packages/react-native-executorch/src/modules/computer_vision/InstanceSegmentationModule.ts
Outdated
Show resolved
Hide resolved
docs/docs/04-typescript-api/02-computer-vision/SemanticSegmentationModule.md
Outdated
Show resolved
Hide resolved
docs/docs/04-typescript-api/02-computer-vision/SemanticSegmentationModule.md
Outdated
Show resolved
Hide resolved
docs/docs/03-hooks/02-computer-vision/useSemanticSegmentation.md
Outdated
Show resolved
Hide resolved
...ive-executorch/common/rnexecutorch/models/instance_segmentation/BaseInstanceSegmentation.cpp
Show resolved
Hide resolved
...ive-executorch/common/rnexecutorch/models/instance_segmentation/BaseInstanceSegmentation.cpp
Outdated
Show resolved
Hide resolved
...ive-executorch/common/rnexecutorch/models/instance_segmentation/BaseInstanceSegmentation.cpp
Outdated
Show resolved
Hide resolved
...ive-executorch/common/rnexecutorch/models/instance_segmentation/BaseInstanceSegmentation.cpp
Outdated
Show resolved
Hide resolved
...ive-executorch/common/rnexecutorch/models/instance_segmentation/BaseInstanceSegmentation.cpp
Outdated
Show resolved
Hide resolved
...ive-executorch/common/rnexecutorch/models/instance_segmentation/BaseInstanceSegmentation.cpp
Outdated
Show resolved
Hide resolved
...ive-executorch/common/rnexecutorch/models/instance_segmentation/BaseInstanceSegmentation.cpp
Outdated
Show resolved
Hide resolved
|
There are some broken links in documentation, can you fix them? |
...ive-executorch/common/rnexecutorch/models/instance_segmentation/BaseInstanceSegmentation.cpp
Outdated
Show resolved
Hide resolved
...ive-executorch/common/rnexecutorch/models/instance_segmentation/BaseInstanceSegmentation.cpp
Outdated
Show resolved
Hide resolved
...ive-executorch/common/rnexecutorch/models/instance_segmentation/BaseInstanceSegmentation.cpp
Show resolved
Hide resolved
...ive-executorch/common/rnexecutorch/models/instance_segmentation/BaseInstanceSegmentation.cpp
Outdated
Show resolved
Hide resolved
...ive-executorch/common/rnexecutorch/models/instance_segmentation/BaseInstanceSegmentation.cpp
Outdated
Show resolved
Hide resolved
...ive-executorch/common/rnexecutorch/models/instance_segmentation/BaseInstanceSegmentation.cpp
Outdated
Show resolved
Hide resolved
docs/docs/04-typescript-api/02-computer-vision/InstanceSegmentationModule.md
Show resolved
Hide resolved
docs/docs/04-typescript-api/02-computer-vision/InstanceSegmentationModule.md
Show resolved
Hide resolved
packages/react-native-executorch/src/modules/computer_vision/InstanceSegmentationModule.ts
Outdated
Show resolved
Hide resolved
packages/react-native-executorch/src/modules/computer_vision/InstanceSegmentationModule.ts
Show resolved
Hide resolved
8557eba to
a755a61
Compare
...ive-executorch/common/rnexecutorch/models/instance_segmentation/BaseInstanceSegmentation.cpp
Outdated
Show resolved
Hide resolved
packages/react-native-executorch/common/rnexecutorch/utils/computer_vision/Processing.cpp
Show resolved
Hide resolved
packages/react-native-executorch/common/rnexecutorch/utils/computer_vision/Processing.h
Outdated
Show resolved
Hide resolved
b28847b to
836c7a2
Compare
a755a61 to
b2fa3be
Compare
c19137a to
b722ff9
Compare
b722ff9 to
0e90871
Compare
...ive-executorch/common/rnexecutorch/models/instance_segmentation/BaseInstanceSegmentation.cpp
Outdated
Show resolved
Hide resolved
...ive-executorch/common/rnexecutorch/models/instance_segmentation/BaseInstanceSegmentation.cpp
Outdated
Show resolved
Hide resolved
|
I will look at this PR in a minute |
55e4265 to
3dd47a5
Compare
68d259b to
c6b21de
Compare
c6b21de to
e4d7a61
Compare
| return VisionModel::modelInputSize(); | ||
| } | ||
| const auto &shape = inputShapes[0]; | ||
| return cv::Size(shape[shape.size() - 2], shape[shape.size() - 1]); |
There was a problem hiding this comment.
| return cv::Size(shape[shape.size() - 2], shape[shape.size() - 1]); | |
| return {shape[shape.size() - 2], shape[shape.size() - 1]}; |
| std::tuple<utils::computer_vision::BBox, float, int32_t> | ||
| BaseInstanceSegmentation::extractDetectionData(const float *bboxData, | ||
| const float *scoresData, | ||
| int32_t index) { | ||
| utils::computer_vision::BBox bbox{ | ||
| bboxData[index * 4], bboxData[index * 4 + 1], bboxData[index * 4 + 2], | ||
| bboxData[index * 4 + 3]}; | ||
| float score = scoresData[index * 2]; | ||
| int32_t label = static_cast<int32_t>(scoresData[index * 2 + 1]); | ||
|
|
||
| return {bbox, score, label}; | ||
| } |
There was a problem hiding this comment.
Why do we pass tuple here, these should be 3 separate function, each fetching different type of data.
| int32_t my2 = | ||
| std::min(maskSize.height, static_cast<int32_t>(std::ceil(my2F))); | ||
|
|
||
| return cv::Rect(mx1, my1, mx2 - mx1, my2 - my1); |
There was a problem hiding this comment.
| return cv::Rect(mx1, my1, mx2 - mx1, my2 - my1); | |
| return {mx1, my1, mx2 - mx1, my2 - my1}; |
| int32_t x2 = std::min(maskSize.width, rect.x + rect.width + 1); | ||
| int32_t y2 = std::min(maskSize.height, rect.y + rect.height + 1); | ||
|
|
||
| return cv::Rect(x1, y1, x2 - x1, y2 - y1); |
There was a problem hiding this comment.
| return cv::Rect(x1, y1, x2 - x1, y2 - y1); | |
| return {x1, y1, x2 - x1, y2 - y1}; |
| TEST(InstanceSegGenerateTests, InvalidImagePathThrows) { | ||
| BaseInstanceSegmentation model(kValidInstanceSegModelPath, {}, {}, true, | ||
| nullptr); | ||
| EXPECT_THROW((void)model.generateFromString("nonexistent_image.jpg", 0.5, 0.5, | ||
| 100, {}, true, kMethodName), |
There was a problem hiding this comment.
These tests are not triggered by the script test
also I get some errors which doesn't occured earlier:
/Users/msluszniak/test_bare_rn/react-native-executorch/packages/react-native-executorch/common/rnexecutorch/tests/integration/TextToSpeechTest.cpp:77: Failure
Expected: Kokoro(kValidLang, kValidTaggerPath, kValidPhonemizerPath, kValidDurationPath, kValidSynthesizerPath, "nonexistent_voice.bin", nullptr) throws an exception of type RnExecutorchError.
Actual: it throws std::invalid_argument with description "File not found: kokoro_en_tagger.json".
[ FAILED ] TTSCtorTests.InvalidVoicePathThrows (0 ms)
This one was fixed with LLM & TTS integration
/Users/msluszniak/test_bare_rn/react-native-executorch/packages/react-native-executorch/common/rnexecutorch/tests/integration/BaseModelTests.h:121: Failure
Type parameterized test suite CommonModelTest is defined via REGISTER_TYPED_TEST_SUITE_P, but never instantiated via INSTANTIATE_TYPED_TEST_SUITE_P. None of the test cases will run.
Ideally, TYPED_TEST_P definitions should only ever be included as part of binaries that intend to use them. (As opposed to, for example, being placed in a library that may be linked in to get other utilities.)
To suppress this error for this test suite, insert the following line (in a non-header) in the namespace it is defined in:
GTEST_ALLOW_UNINSTANTIATED_PARAMETERIZED_TEST(CommonModelTest);
[ FAILED ] GoogleTestVerification.UninstantiatedTypeParameterizedTestSuite (0 ms)
/Users/msluszniak/test_bare_rn/react-native-executorch/packages/react-native-executorch/common/rnexecutorch/tests/integration/BaseModelTests.h:108: Failure
Expected: Traits::callGenerate(model) doesn't throw an exception.
Actual: it throws rnexecutorch::RnExecutorchError with description "The model's forward function did not succeed. Ensure the model input is correct.".
/Users/msluszniak/test_bare_rn/react-native-executorch/packages/react-native-executorch/common/rnexecutorch/tests/integration/BaseModelTests.h:109: Failure
Expected: Traits::callGenerate(model) doesn't throw an exception.
Actual: it throws rnexecutorch::RnExecutorchError with description "The model's forward function did not succeed. Ensure the model input is correct.".
/Users/msluszniak/test_bare_rn/react-native-executorch/packages/react-native-executorch/common/rnexecutorch/tests/integration/BaseModelTests.h:110: Failure
Expected: Traits::callGenerate(model) doesn't throw an exception.
Actual: it throws rnexecutorch::RnExecutorchError with description "The model's forward function did not succeed. Ensure the model input is correct.".
[ FAILED ] TextToImage/CommonModelTest/0.MultipleGeneratesWork, where TypeParam = rnexecutorch::models::text_to_image::TextToImage (4981 ms)
| @@ -0,0 +1,277 @@ | |||
| import Spinner from '../../components/Spinner'; | |||
There was a problem hiding this comment.
There are some thing to fix in demo app:
- Why there is object detection live when there is a separate vision camera screen, also when clicking on this, I got unmatched route.
- Why instance and semantic segmentation are not next to each other on the list below
- There is no way to get back from vision camera example, I missed it in previous review, cc: @NorbertKlockiewicz
There was a problem hiding this comment.
- You can just swipe from the left side
There was a problem hiding this comment.
@NorbertKlockiewicz I don't thing that is good enough. If I had a problem and haven't thought about that swipe, then potentially many more users might have the same problem (there is no intuition that there might be something here). That's my opinion.

Description
This PR introduces Instance Segmentation Module. Provided API allows for using two predefined models (RF-DETR and YOLO26-Seg family) as well as plugging custom models through fromCustomConfig.
On top of adding new API, this PR adds common CV utilities on the CPP side, as well as migrates Object Detection to leverage them.
Introduces a breaking change?
Type of change
Tested on
Testing instructions
Screenshots
Related issues
#825
Additional notes