Upload annotated dataset via Python API raises AttributeError but upload via WebUI works fine

[gpadiolleau]

I'm trying to upload a dataset via Python API with annotations file in JSON.

But the upload fails after one image is uploaded with an Python AttributeError:

File "/home/user/anaconda3/envs/py310_roboflow/lib/python3.10/site-packages/roboflow/core/project.py", line 451, in upload
    self.single_upload(
  File "/home/user/anaconda3/envs/py310_roboflow/lib/python3.10/site-packages/roboflow/core/project.py", line 581, in single_upload
    uploaded_annotation, annotation_time = self.save_annotation(
  File "/home/user/anaconda3/envs/py310_roboflow/lib/python3.10/site-packages/roboflow/core/project.py", line 522, in save_annotation
    annotation = rfapi.save_annotation(
  File "/home/user/anaconda3/envs/py310_roboflow/lib/python3.10/site-packages/roboflow/adapters/rfapi.py", line 169, in save_annotation
    raise _save_annotation_error(response)
  File "/home/user/anaconda3/envs/py310_roboflow/lib/python3.10/site-packages/roboflow/adapters/rfapi.py", line 239, in _save_annotation_error
    if err_msg.get("message"):
AttributeError: 'str' object has no attribute 'get'

It appears that the error message returned by API call is not handled, since the real API response is:

responsejson={'error': 'Unknown error'}
err_msg='Unknown error'

When I try to upload the annotated dataset via the WebUI, it works fine.
How can I have further information about the error ?

Edit: I followed step-by-step the tutorial (https://docs.roboflow.com/api-reference/images/upload-an-annotation) to find if I'm doing something wrong, but get the exact same error.

[venkatram-dev]

The tutorial works for me in python 3.10 and 3.11 as well.

Please try upgrading your roboflow install.

If it still doesn't work, please provide sample images , annotations and exact code snippet used.

[gpadiolleau]

I upgraded roboflow from 1.1.43 to 1.1.44, but still get the same error.

Images are simple RGB .jpg images of maximum size about ~160ko.

The following code produces the error (it is the tutorial):

import glob
from roboflow import Roboflow

# Initialize Roboflow client
rf = Roboflow()

# Directory path and file extension for images
dir_name = "/home/user/s3buckets/datasets/chunked_dataset/sdcw/2024-08-28-095829/images"
file_extension_type = ".jpg"

# Annotation file path and format (e.g., .coco.json)
annotation_filename = "/home/user/s3buckets/datasets/chunked_dataset/sdcw/2024-08-28-095829/annotations/my_project/_annotations.coco.json"

# Get the upload project from Roboflow workspace
project = rf.workspace().project("my-project")

# Upload images
image_glob = glob.glob(dir_name + '/*' + file_extension_type)
for image_path in image_glob:
    print(project.single_upload(
        image_path=image_path,
        annotation_path=annotation_filename,
    ))

Part of the _annotations.coco.json file:

{
    "info": {
        "year": "2024",
        "contributor": "",
        "date_created": "2024-09-02T12:57:20+00:00"
    },
    "licenses": [
        {
            "id": 1,
            "url": "",
            "name": "Private"
        }
    ],
    "categories": [
        {
            "id": 0,
            "name": "cardboard-universal",
            "supercategory": "none"
        },
        {
            "id": 1,
            "name": "universal",
            "supercategory": "cardboard-universal"
        },
        {
            "id": 2,
            "name": "cardboard",
            "supercategory": "cardboard-universal"
        }
    ],
    "images": [
        {
            "id": 0,
            "license": 1,
            "file_name": "a7640d18-ff18-4294-be6e-ef30d01f016e.jpg",
            "height": 640,
            "width": 640,
            "date_captured": "2024-09-02T12:57:20+00:00"
        },
        {
            "id": 1,
            "license": 1,
            "file_name": "0ef1bf8d-cee1-482b-a71f-7d544413e154.jpg",
            "height": 640,
            "width": 640,
            "date_captured": "2024-09-02T12:57:20+00:00"
        },
        ...
    ],
    "annotations": [
        {
            "image_id": 0,
            "id": 0,
            "category_id": 1,
            "bbox": [
                70,
                448,
                41.25,
                157.5
            ],
            "area": 6496.875,
            "segmentation": [
                [
                    92.5,
                    604.375,
                    111.25,
                    598.125,
                    95.625,
                    447.5,
                    70,
                    454.375,
                    82.5,
                    600.625,
                    84.375,
                    605
                ]
            ],
            "iscrowd": 0
        },
        {
            "image_id": 0,
            "id": 1,
            "category_id": 1,
            "bbox": [
                69,
                295,
                88.75,
                128.125
            ],
            "area": 11371.094,
            "segmentation": [
                [
                    147.5,
                    423.125,
                    148.75,
                    376.875,
                    155,
                    348.125,
                    157.5,
                    300.625,
                    146.875,
                    295,
                    123.125,
                    296.25,
                    118.75,
                    313.125,
                    90,
                    365.625,
                    78.75,
                    366.875,
                    68.75,
                    418.125
                ]
            ],
            "iscrowd": 0
        },
        ...
    ]
}

I thought about something: some images in the images folder does not have related annotations in the file, can this cause the problem ?
And again what I don't understand is why it is working well when using the WebUI.

Thanks for your help.

[gpadiolleau]

As additional information:
I downloaded a Roboflow dataset with python API and then tried to upload it to another new fresh project with Python API and still get the same error.

[venkatram-dev]

yeah Interesting. I tried to replicate the error you get and was able to.

1. Removed some ids from images array in the annotation json. But that gave a different error.
   let us leave that aside for now.

2. I corrupted the _annotations.coco.json and I am able to replicate the error.

correct annotation

"images":[{"id":0,"license":1,"file_name":"f52_jpg.rf.b74946257dcd3493e38b86a9bc0fa425.jpg","height":1024,"width":1792,"date_captured":"2024-08-23T22:05:42+00:00"},{"id":1,"license":1,"file_name":"f34_jpg.rf.d84ad1992847d85e22f0dfc1b3d27392.jpg","height":1024,"width":1792,"date_captured":"2024-08-23T22:05:42+00:00"},
...
]

Incorrect annotation

I manually modified the id for the second element to be zero instead of one.

"images":[{"id":0,"license":1,"file_name":"f52_jpg.rf.b74946257dcd3493e38b86a9bc0fa425.jpg","height":1024,"width":1792,"date_captured":"2024-08-23T22:05:42+00:00"},{"id":0,"license":1,"file_name":"f34_jpg.rf.d84ad1992847d85e22f0dfc1b3d27392.jpg","height":1024,"width":1792,"date_captured":"2024-08-23T22:05:42+00:00"},
...
]

Now, when I run the python code i get below error.

status code 500
responsejson {'error': 'Unknown error'}

    if err_msg.get("message"):
       ^^^^^^^^^^^
AttributeError: 'str' object has no attribute 'get'

Also, attached is the screenshot from web UI, where the file with incorrect annotation went to not annotated tab.
There were no errors in the UI.

https://app.roboflow.com/test-4h9f7/test-upload-nfbt0/annotate

We need to debug the code for web app/ API endpoints to understand what is going on and why its returning 500/Unknown error'

I am not able to find the code the web app/ API endpoints for annotate.

Do you know where it could be?

Meanwhile you might want to clean up your annotation file for any errors.(since we don't know which annotations cause issue, you might need to do manual steps by trying few images at a time :) )

[gpadiolleau]

Unfortunately, I ran a simple script to check if any image ids or annotation ids are duplicate (looking for the case you described) but I have no duplicates in my file.

As additional information: I downloaded a Roboflow dataset with python API and then tried to upload it to another new fresh project with Python API and still get the same error.

I am not sure this is the real problem I have since I tried to just download a dataset and then upload it to another clean new fresh project and got the same error.
Maybe the two issues are related.

I am not able to find the code the web app/ API endpoints for annotate.

Do you know where it could be?

No i didnt check this. I didnt know it was os too, i'll try to find it if i have time to.

[gpadiolleau]

Hey @venkatram-dev, I'm coming with news !

It seems you changed something because the "unknown error" disappeared, that's good news !
However, I now get a new error (more meaningful!) when I try to upload through python API:
roboflow.adapters.rfapi.AnnotationSaveError: Image was already annotated.

I checked all my code (seems ok), generate new annotations with and checked if any duplicates in image ids, image names or annotation ids but the script found nothing.

However, I know that images and annotations with same name where previously uploaded (deleted since) to the project. Can i be caused by a conflict with images/annotations that where previously on the project ?

[venkatram-dev]

Hi,

I am not from roboflow team. Looks like the api code is not open source, so I did not change anything :) .
Not sure if roboflow team changed something.

Anyways, regarding your latest error "roboflow.adapters.rfapi.AnnotationSaveError: Image was already annotated."

Can you please try annotation_overwrite=True option. Below code worked for me.

`# Upload images
image_glob = glob.glob(dir_name + '/*' + file_extension_type)
for image_path in image_glob:
    print(project.single_upload(
        image_path=image_path,
        annotation_path=annotation_filename,
        annotation_overwrite=True
    ))`

[gpadiolleau]

Hi,

Haha, i thought you were !

Nice, thanks for the tip annotation_overwrite=True seems to work !