SQL Server Connector | GRPC Error - Failed to connect to all addresses

[jessepasswaters]

While executing the connector, I am able to connect to SQL Server and it appears that it is able to do the initial auth/connection to the DataCatalog API, but once it starts performing actions, it fails.

I have confirmed that the service account that I am using has DataCatalog owner permissions and that the environment is configured properly.

With DataCatalog being rolled into DataPlex, I wasn't sure if there were additional changes required as part of that?

Error received:

INFO:root:
162 tables ready to be ingested...
INFO:root:
==============Ingest metadata===============
DEBUG:google.auth._default:Checking /home/user/code/keys/exp-metadata-dev-20221010-7ab62fda6fc4.json for explicit credentials as part of auth process...
INFO:root:
INFO:root:Starting to clean up the catalog...
Traceback (most recent call last):
  File "/home/user/code/datacatalog_sqlserver/datacatalog/lib/python3.9/site-packages/google/api_core/grpc_helpers.py", line 72, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/home/user/code/datacatalog_sqlserver/datacatalog/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/home/user/code/datacatalog_sqlserver/datacatalog/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "failed to connect to all addresses; last error: UNKNOWN: ipv6:%5B2607:f8b0:4009:806::200a%5D:443: Failed to connect to remote host: Connection refused"
        debug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv6:%5B2607:f8b0:4009:806::200a%5D:443: Failed to connect to remote host: Connection refused {created_time:"2022-10-26T12:55:37.9562088-04:00", grpc_status:14}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/user/code/datacatalog_sqlserver/datacatalog/bin/google-datacatalog-sqlserver-connector", line 8, in <module>
    sys.exit(main())
  File "/home/user/code/datacatalog_sqlserver/datacatalog/lib/python3.9/site-packages/google/datacatalog_connectors/sqlserver/datacatalog_cli.py", line 105, in main
    SQLServer2DatacatalogCli().run(argv[1:] if len(argv) > 0 else argv)
  File "/home/user/code/datacatalog_sqlserver/datacatalog/lib/python3.9/site-packages/google/datacatalog_connectors/rdbms/datacatalog_cli.py", line 46, in run
    self._get_datacatalog_synchronizer()(
  File "/home/user/code/datacatalog_sqlserver/datacatalog/lib/python3.9/site-packages/google/datacatalog_connectors/rdbms/sync/datacatalog_synchronizer.py", line 105, in run
    self.__delete_obsolete_metadata(prepared_entries)
  File "/home/user/code/datacatalog_sqlserver/datacatalog/lib/python3.9/site-packages/google/datacatalog_connectors/rdbms/sync/datacatalog_synchronizer.py", line 161, in __delete_obsolete_metadata
    cleaner.delete_obsolete_metadata(
  File "/home/user/code/datacatalog_sqlserver/datacatalog/lib/python3.9/site-packages/google/datacatalog_connectors/commons/cleanup/datacatalog_metadata_cleaner.py", line 55, in delete_obsolete_metadata
    self.__datacatalog_facade.search_catalog_relative_resource_name(
  File "/home/user/code/datacatalog_sqlserver/datacatalog/lib/python3.9/site-packages/google/datacatalog_connectors/commons/datacatalog_facade.py", line 499, in search_catalog_relative_resource_name
    for result in self.search_catalog(query)
  File "/home/user/code/datacatalog_sqlserver/datacatalog/lib/python3.9/site-packages/google/datacatalog_connectors/commons/datacatalog_facade.py", line 487, in search_catalog
    result for result in self.__datacatalog.search_catalog(request)
  File "/home/user/code/datacatalog_sqlserver/datacatalog/lib/python3.9/site-packages/google/cloud/datacatalog_v1beta1/services/data_catalog/client.py", line 459, in search_catalog
    response = rpc(
  File "/home/user/code/datacatalog_sqlserver/datacatalog/lib/python3.9/site-packages/google/api_core/gapic_v1/method.py", line 154, in __call__
    return wrapped_func(*args, **kwargs)
  File "/home/user/code/datacatalog_sqlserver/datacatalog/lib/python3.9/site-packages/google/api_core/grpc_helpers.py", line 74, in error_remapped_callable
    raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.ServiceUnavailable: 503 failed to connect to all addresses; last error: UNKNOWN: ipv6:%5B2607:f8b0:4009:806::200a%5D:443: Failed to connect to remote host: Connection refused

Current installed packages:

$ python3 -m pip install -r requirements.txt --upgrade
Requirement already satisfied: aiohttp>=3.8.3 in ./datacatalog/lib/python3.9/site-packages (from -r requirements.txt (line 1)) (3.8.3)
Requirement already satisfied: aiosignal>=1.2.0 in ./datacatalog/lib/python3.9/site-packages (from -r requirements.txt (line 2)) (1.2.0)
Requirement already satisfied: async-timeout>=4.0.2 in ./datacatalog/lib/python3.9/site-packages (from -r requirements.txt (line 3)) (4.0.2)
Requirement already satisfied: attrs>=22.1.0 in ./datacatalog/lib/python3.9/site-packages (from -r requirements.txt (line 4)) (22.1.0)
Requirement already satisfied: cachetools>=5.2.0 in ./datacatalog/lib/python3.9/site-packages (from -r requirements.txt (line 5)) (5.2.0)
Requirement already satisfied: certifi>=2022.9.24 in ./datacatalog/lib/python3.9/site-packages (from -r requirements.txt (line 6)) (2022.9.24)
Requirement already satisfied: charset-normalizer>=2.1.1 in ./datacatalog/lib/python3.9/site-packages (from -r requirements.txt (line 7)) (2.1.1)
Collecting charset-normalizer>=2.1.1
  Using cached charset_normalizer-3.0.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (198 kB)
Requirement already satisfied: contextlib2>=21.6.0 in ./datacatalog/lib/python3.9/site-packages (from -r requirements.txt (line 8)) (21.6.0)
Requirement already satisfied: decorator>=5.1.1 in ./datacatalog/lib/python3.9/site-packages (from -r requirements.txt (line 9)) (5.1.1)
Requirement already satisfied: frozenlist>=1.3.1 in ./datacatalog/lib/python3.9/site-packages (from -r requirements.txt (line 10)) (1.3.1)
Requirement already satisfied: fsspec>=2022.10.0 in ./datacatalog/lib/python3.9/site-packages (from -r requirements.txt (line 11)) (2022.10.0)
Requirement already satisfied: gcsfs>=2022.10.0 in ./datacatalog/lib/python3.9/site-packages (from -r requirements.txt (line 12)) (2022.10.0)
Requirement already satisfied: google-api-core>=2.10.2 in ./datacatalog/lib/python3.9/site-packages (from -r requirements.txt (line 13)) (2.10.2)
Requirement already satisfied: google-auth>=2.13.0 in ./datacatalog/lib/python3.9/site-packages (from -r requirements.txt (line 14)) (2.13.0)
Requirement already satisfied: google-auth-oauthlib>=0.7.0 in ./datacatalog/lib/python3.9/site-packages (from -r requirements.txt (line 15)) (0.7.0)
Requirement already satisfied: google-cloud-core>=2.3.2 in ./datacatalog/lib/python3.9/site-packages (from -r requirements.txt (line 16)) (2.3.2)
Requirement already satisfied: google-cloud-datacatalog>=2.0.2 in ./datacatalog/lib/python3.9/site-packages (from -r requirements.txt (line 17)) (2.0.2)
Collecting google-cloud-datacatalog>=2.0.2
  Using cached google_cloud_datacatalog-3.9.3-py2.py3-none-any.whl (300 kB)
Requirement already satisfied: google-cloud-monitoring>=1.1.2 in ./datacatalog/lib/python3.9/site-packages (from -r requirements.txt (line 18)) (1.1.2)
Collecting google-cloud-monitoring>=1.1.2
  Using cached google_cloud_monitoring-2.11.3-py2.py3-none-any.whl (286 kB)
Requirement already satisfied: google-cloud-storage>=2.5.0 in ./datacatalog/lib/python3.9/site-packages (from -r requirements.txt (line 19)) (2.5.0)
Requirement already satisfied: google-crc32c>=1.5.0 in ./datacatalog/lib/python3.9/site-packages (from -r requirements.txt (line 20)) (1.5.0)
Requirement already satisfied: google-datacatalog-connectors-commons>=0.6.9 in ./datacatalog/lib/python3.9/site-packages (from -r requirements.txt (line 21)) (0.6.9)
Requirement already satisfied: google-datacatalog-rdbms-connector>=0.12.2 in ./datacatalog/lib/python3.9/site-packages (from -r requirements.txt (line 22)) (0.12.2)
Requirement already satisfied: google-datacatalog-sqlserver-connector>=0.10.0 in ./datacatalog/lib/python3.9/site-packages (from -r requirements.txt (line 23)) (0.10.0)
Requirement already satisfied: google-resumable-media>=2.4.0 in ./datacatalog/lib/python3.9/site-packages (from -r requirements.txt (line 24)) (2.4.0)
Requirement already satisfied: googleapis-common-protos>=1.56.4 in ./datacatalog/lib/python3.9/site-packages (from -r requirements.txt (line 25)) (1.56.4)
Requirement already satisfied: grpc-google-iam-v1>=0.12.4 in ./datacatalog/lib/python3.9/site-packages (from -r requirements.txt (line 26)) (0.12.4)
Requirement already satisfied: grpcio>=1.50.0 in ./datacatalog/lib/python3.9/site-packages (from -r requirements.txt (line 27)) (1.50.0)
Requirement already satisfied: grpcio-status>=1.48.2 in ./datacatalog/lib/python3.9/site-packages (from -r requirements.txt (line 28)) (1.48.2)
Collecting grpcio-status>=1.48.2
  Using cached grpcio_status-1.50.0-py3-none-any.whl (14 kB)
Requirement already satisfied: idna>=3.4 in ./datacatalog/lib/python3.9/site-packages (from -r requirements.txt (line 29)) (3.4)
Requirement already satisfied: libcst>=0.4.7 in ./datacatalog/lib/python3.9/site-packages (from -r requirements.txt (line 30)) (0.4.7)
Requirement already satisfied: multidict>=6.0.2 in ./datacatalog/lib/python3.9/site-packages (from -r requirements.txt (line 31)) (6.0.2)
Requirement already satisfied: mypy-extensions>=0.4.3 in ./datacatalog/lib/python3.9/site-packages (from -r requirements.txt (line 32)) (0.4.3)
Requirement already satisfied: numpy>=1.23.4 in ./datacatalog/lib/python3.9/site-packages (from -r requirements.txt (line 33)) (1.23.4)
Requirement already satisfied: oauthlib>=3.2.2 in ./datacatalog/lib/python3.9/site-packages (from -r requirements.txt (line 34)) (3.2.2)
Requirement already satisfied: pandas>=1.1.5 in ./datacatalog/lib/python3.9/site-packages (from -r requirements.txt (line 35)) (1.1.5)
Collecting pandas>=1.1.5
  Using cached pandas-1.5.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.2 MB)
Requirement already satisfied: proto-plus>=1.22.1 in ./datacatalog/lib/python3.9/site-packages (from -r requirements.txt (line 36)) (1.22.1)
Requirement already satisfied: protobuf>=3.20.3 in ./datacatalog/lib/python3.9/site-packages (from -r requirements.txt (line 37)) (3.20.3)
Collecting protobuf>=3.20.3
  Using cached protobuf-4.21.8-cp37-abi3-manylinux2014_x86_64.whl (408 kB)
Requirement already satisfied: pyasn1>=0.4.8 in ./datacatalog/lib/python3.9/site-packages (from -r requirements.txt (line 38)) (0.4.8)
Requirement already satisfied: pyasn1-modules>=0.2.8 in ./datacatalog/lib/python3.9/site-packages (from -r requirements.txt (line 39)) (0.2.8)
Requirement already satisfied: pyodbc>=4.0.34 in ./datacatalog/lib/python3.9/site-packages (from -r requirements.txt (line 40)) (4.0.34)
Requirement already satisfied: python-dateutil>=2.8.2 in ./datacatalog/lib/python3.9/site-packages (from -r requirements.txt (line 41)) (2.8.2)
Requirement already satisfied: pytz>=2022.5 in ./datacatalog/lib/python3.9/site-packages (from -r requirements.txt (line 42)) (2022.5)
Requirement already satisfied: PyYAML>=6.0 in ./datacatalog/lib/python3.9/site-packages (from -r requirements.txt (line 43)) (6.0)
Requirement already satisfied: requests>=2.28.1 in ./datacatalog/lib/python3.9/site-packages (from -r requirements.txt (line 44)) (2.28.1)
Requirement already satisfied: requests-oauthlib>=1.3.1 in ./datacatalog/lib/python3.9/site-packages (from -r requirements.txt (line 45)) (1.3.1)
Requirement already satisfied: rsa>=4.9 in ./datacatalog/lib/python3.9/site-packages (from -r requirements.txt (line 46)) (4.9)
Requirement already satisfied: schema>=0.7.5 in ./datacatalog/lib/python3.9/site-packages (from -r requirements.txt (line 47)) (0.7.5)
Requirement already satisfied: six>=1.16.0 in ./datacatalog/lib/python3.9/site-packages (from -r requirements.txt (line 48)) (1.16.0)
Requirement already satisfied: typing-inspect>=0.8.0 in ./datacatalog/lib/python3.9/site-packages (from -r requirements.txt (line 49)) (0.8.0)
Requirement already satisfied: typing_extensions>=4.4.0 in ./datacatalog/lib/python3.9/site-packages (from -r requirements.txt (line 50)) (4.4.0)
Requirement already satisfied: urllib3>=1.26.12 in ./datacatalog/lib/python3.9/site-packages (from -r requirements.txt (line 51)) (1.26.12)
Requirement already satisfied: yarl>=1.8.1 in ./datacatalog/lib/python3.9/site-packages (from -r requirements.txt (line 52)) (1.8.1)
Collecting grpcio-status>=1.48.2
  Using cached grpcio_status-1.49.1-py3-none-any.whl (14 kB)

Thanks in advance for any help that you can provide.