Bump botocore dependency specification

[jakob-keller]

Description of Change

This PR intends to improve general compatibility of aiobotocore within the Python ecosystem by bumping the dependency specification of botocore, as well as boto3 and awscli.

Assumptions

Upstream diff contains changes that require adjustments to the aiobotocore codebase.

Checklist for All Submissions

• I have added change info to CHANGES.rst
• If this is resolving an issue (needed so future developers can determine if change is still necessary and under what conditions) (can be provided via link to issue with these details): closes Bump botocore dependency #1229
  • Detailed description of issue
  • Alternative methods considered (if any)
  • How issue is being resolved
  • How issue can be reproduced
• If this is providing a new feature (can be provided via link to issue with these details):
  • Detailed description of new feature
  • Why needed
  • Alternatives methods considered (if any)

Checklist when updating botocore and/or aiohttp versions

• I have read and followed CONTRIBUTING.rst
• I have updated test_patches.py where/if appropriate (also check if no changes necessary)
• I have ensured that the awscli/boto3 versions match the updated botocore version

[codecov]

Codecov Report

Attention: Patch coverage is 33.33333% with 16 lines in your changes missing coverage. Please review.

Project coverage is 86.05%. Comparing base (e7d6ddf) to head (12b7fc3).

Files with missing lines	Patch %	Lines
aiobotocore/signers.py	25.00%	15 Missing ⚠️
aiobotocore/handlers.py	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1221      +/-   ##
==========================================
- Coverage   86.16%   86.05%   -0.12%     
==========================================
  Files          64       64              
  Lines        5986     5995       +9     
==========================================
+ Hits         5158     5159       +1     
- Misses        828      836       +8     

Flag	Coverage Δ
unittests	86.05% <33.33%> (-0.12%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[jakob-keller]

Notes to reviewers:

• I believe that upstream does not affect any codepaths relevant for us, beyond what is included in this PR. However, I am not 100% sure about _handle_200_error() in handlers.py. Please take a closer look at that aspect, if you can.
• FWIW: test coverage is not actually any worse than it was before.

[thehesiod]

Notes to reviewers:

• I believe that upstream does not affect any codepaths relevant for us, beyond what is included in this PR. However, I am not 100% sure about _handle_200_error() in handlers.py. Please take a closer look at that aspect, if you can.
• FWIW: test coverage is not actually any worse than it was before.

ya I'm not sure if response_dict['body'] is a coroutine or the actual bytes. I'd put a breakpoint and see what it looks like

[thehesiod]

I think it's populated here:

aiobotocore/aiobotocore/endpoint.py

Lines 57 to 65 in 8a50b61

	if response_dict['status_code'] >= 300:
	response_dict['body'] = await http_response.content
	elif operation_model.has_event_stream_output:
	response_dict['body'] = http_response.raw
	elif operation_model.has_streaming_output:
	length = response_dict['headers'].get('content-length')
	response_dict['body'] = StreamingBody(http_response.raw, length)
	else:
	response_dict['body'] = await http_response.content

so I think it depends on the response type

[jakob-keller]

I think it's populated here:

aiobotocore/aiobotocore/endpoint.py

Lines 57 to 65 in 8a50b61

	if response_dict['status_code'] >= 300:
	response_dict['body'] = await http_response.content
	elif operation_model.has_event_stream_output:
	response_dict['body'] = http_response.raw
	elif operation_model.has_streaming_output:
	length = response_dict['headers'].get('content-length')
	response_dict['body'] = StreamingBody(http_response.raw, length)
	else:
	response_dict['body'] = await http_response.content

so I think it depends on the response type

Exactly: Case 1 and 4 result in bytes, case 3 cannot be reached in this context, but case 2 on line 60 is tricky. I have no idea under what condition this is reached, if at all, and whether it results in something that needs to be awaited. Beats me!

[thehesiod]

I think here's an example: https://botocore.amazonaws.com/v1/documentation/api/latest/reference/eventstream.html

[thehesiod]

isn't line 63 also a coroutine? That's like a s3 get-object body i think

[thehesiod]

ideally we could add tests for this with moto

[jakob-keller]

isn't line 63 also a coroutine? That's like a s3 get-object body i think

There's a guard function _should_handle_200_error() that makes that codepath impossible to reach, as far as I can tell.

[jakob-keller]

I think here's an example: https://botocore.amazonaws.com/v1/documentation/api/latest/reference/eventstream.html

ideally we could add tests for this with moto

I could give it a shot towards the end of the week. Thanks for looking into it!

[jakob-keller]

isn't line 63 also a coroutine? That's like a s3 get-object body i think

There's a guard function _should_handle_200_error() that makes that codepath impossible to reach, as far as I can tell.

Actually, I am suspecting that this guard function also prevents case 2 / line 60 from being reached.

[jakob-keller]

I've included all pending botocore updates into this PR, including support for DSQL auth and all the other new and updated AWS services.

We still need include unit tests, in particular with regards to the issue discussed above. Hopefully this weekend...

[jakob-keller]

I think here's an example: https://botocore.amazonaws.com/v1/documentation/api/latest/reference/eventstream.html

Well, that example doesn't work for me: AWS no longer accepts new users for S3 Select. I will need to find another service that uses event streams.

[thehesiod]

looks like bedrock supports eventstreams as well: https://docs.aws.amazon.com/bedrock/latest/userguide/agents-test.html

[thehesiod]

stream example: https://stackoverflow.com/questions/78378549/botocore-eventstream-invalidheaderslength

[thehesiod]

oh duh, we should just port their tests, lemme c if I can do it

[thehesiod]

@jakob-keller see latest change to #1232. Wasn't sure how to push to your branch. If you give me rights can push there or feel free to cherry pick it

[thehesiod]

seems like it's fairly easy to pull their tests in now, perhaps we should try to pull the whole thing over

[jakob-keller]

seems like it's fairly easy to pull their tests in now, perhaps we should try to pull the whole thing over

That sounds fantastic, thanks!

I have to admit I'm still not familiar with our test suite. Wouldn't it make sense to separate your test related work into its own PR and then revisit this PR?