Web connector read timeout

[perfecto25]

wondering if theres a way to increase the read timeout on web connectors (couldnt find it in docs)

If i add a new web connector that searches recursively, I'm getting this error

Failed to fetch 'https://mydomain.company.com/': Unable to reach https://mydomain.company.com/ - check your internet connection: HTTPSConnectionPool(host='mydomain.company.com', port=443): Read timed out. (read timeout=3)

I know its not DNS or firewall, because I added another web connector for same URL but "single" not recursive scrape, and that worked, it indexed 1 document

Recursive is timing out, I think due to the 3 second timeout variable. Anyway to control that 3 sec var from .env or some other config?

Thanks.

[perfecto25]

was able to get it to index a web page by modifiying timeout value inside the container,

def check_internet_connection(url: str) -> None:
    try:
        response = requests.get(url, timeout=3)

changed this to 30

danswer/connectors/web/connector.py

ill add PR to provide web timeout via ENV var