Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Attempt to automatically adjust input image size #10

Open
bruno-f-cruz opened this issue Nov 11, 2022 · 0 comments
Open

Attempt to automatically adjust input image size #10

bruno-f-cruz opened this issue Nov 11, 2022 · 0 comments

Comments

@bruno-f-cruz
Copy link
Contributor

Sleap networks may have different input sizes than the videos they were trained on. This is done to accommodate two requirements:

  1. The image size is divisible by the max stride of the model (usually 16 or 32) so we don't get mismatched tensor sizes across skip connections due to shape rounding/truncation
  2. To ensure that all images in the same batch are of the same size so that we can handle variable image sizes.

This pre-processing step is automatically handled in the SLEAP API but not currently in Bonsai. They do it by padding, with zeros, at the bottom, and right of the image (which means coordinates should not change).
This pre-process step can be easily done in Bonsai by using the ResizeCanvas operator.

For the sake of usability, we could add this optional functionality to the predict nodes. Unfortunately, the training_config.json we are currently loading in bonsai with all the metadata does not have the network input size information.

Few possible solutions:

  1. Load the info.json that results from the export sleap functionality. This would require loading and parsing an additional JSON file.
  2. An alternative to 1. would be to simply merge the two json files in a single output file during the export stage, which would save users the bother to keep track of what metafiles correspond to what networks.
  3. It might be possible to infer the network input size form the deserialized frozen network. Must investigate if this is a valid approach.
@bruno-f-cruz bruno-f-cruz changed the title Add option to attempt to automatically adjust input image size Attempt to automatically adjust input image size Nov 11, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant