Add gradio-tools plugin #14

freddyaboulton · 2023-04-21T20:49:09Z

As discussed in Significant-Gravitas/AutoGPT#1430, adding the gradio-tools module as a plugin.

ntindle · 2023-04-21T21:45:12Z

This is a great start! Would love a few tests to cover some of the code here so that changes won't break this down the line

ntindle · 2023-04-22T17:37:06Z

src/autogpt_plugins/gradio-tools/__init__.py

@@ -0,0 +1,239 @@
+"""Twitter API integrations using Tweepy."""


Fix the docstring

ntindle · 2023-04-22T17:37:36Z

src/autogpt_plugins/gradio-tools/__init__.py

+        self._name = "autogpt-gradio-tools"
+        self._version = "0.1.0"
+        self._description = "Calling Gradio Apps ."
+        ts = [t for t in os.getenv("AUTOGPT_GRADIO_TOOLS", "").split(",") if t != ""]


Check if these exist before loading the plugin commands

you can see how the email plugin does it for context

Awesome - just pushed up this change!

…gins into add-gradio-tools

freddyaboulton · 2023-04-23T23:03:33Z

@ntindle I just added the tests!

ntindle · 2023-04-23T23:03:39Z

I’m ooo today but I’ll try and test/merge this tomorrow

ntindle

Please validate that all commands work

Validate tests as well and include tests for each tool

ntindle · 2023-04-26T06:05:31Z

src/autogpt_plugins/autogpt_gradio_tools/tools.py

+import os
+
+
+WORKSPACE_DIR = (Path(os.getcwd()) / "auto_gpt_workspace").resolve()


@collijk How should this be loaded in a plugin?

ntindle · 2023-04-26T06:10:48Z

src/autogpt_plugins/autogpt_gradio_tools/__init__.py

+if (Path(os.getcwd()) / ".env").exists():
+    with open(str(Path(os.getcwd()) / ".env"), 'r') as fp:
+        load_dotenv(stream=fp)
+


This shouldn't be needed

ntindle · 2023-04-26T06:11:17Z

src/autogpt_plugins/autogpt_gradio_tools/__init__.py

+TOOLS = [
+    AutoGPTStableDiffusion(hf_token=os.getenv("GRADIO_TOOLS_HF_TOKEN")),
+    AutoGPTCaptioner(hf_token=os.getenv("GRADIO_TOOLS_HF_TOKEN")),
+    AutoGPTWhisperTool(hf_token=os.getenv("GRADIO_TOOLS_HF_TOKEN")),
+    AutoGPTTextToVideoTool(hf_token=os.getenv("GRADIO_TOOLS_HF_TOKEN")),
+    AutoGPTPromptGeneratorTool(hf_token=os.getenv("GRADIO_TOOLS_HF_TOKEN")),
+    AutoGPTDocumentAnsweringTool(hf_token=os.getenv("GRADIO_TOOLS_HF_TOKEN")),
+    AutoGPTImageToMusicTool(hf_token=os.getenv("GRADIO_TOOLS_HF_TOKEN")),
+    AutoGPTClipInterrogatorTool(hf_token=os.getenv("GRADIO_TOOLS_HF_TOKEN"))
+]


Only run this if your plugin is loaded. Take a look at the email plugin to see how to do this

ntindle · 2023-04-26T06:13:10Z

src/autogpt_plugins/autogpt_gradio_tools/__init__.py

+TOOLS = [
+    AutoGPTStableDiffusion(hf_token=os.getenv("GRADIO_TOOLS_HF_TOKEN")),
+    AutoGPTCaptioner(hf_token=os.getenv("GRADIO_TOOLS_HF_TOKEN")),
+    AutoGPTWhisperTool(hf_token=os.getenv("GRADIO_TOOLS_HF_TOKEN")),
+    AutoGPTTextToVideoTool(hf_token=os.getenv("GRADIO_TOOLS_HF_TOKEN")),
+    AutoGPTPromptGeneratorTool(hf_token=os.getenv("GRADIO_TOOLS_HF_TOKEN")),
+    AutoGPTDocumentAnsweringTool(hf_token=os.getenv("GRADIO_TOOLS_HF_TOKEN")),
+    AutoGPTImageToMusicTool(hf_token=os.getenv("GRADIO_TOOLS_HF_TOKEN")),
+    AutoGPTClipInterrogatorTool(hf_token=os.getenv("GRADIO_TOOLS_HF_TOKEN"))
+]


don't use this like a global variable, initialize it within the init and scope it to the plugin's class instance

ntindle · 2023-04-26T06:13:40Z

src/autogpt_plugins/autogpt_gradio_tools/__init__.py

+    AutoGPTClipInterrogatorTool(hf_token=os.getenv("GRADIO_TOOLS_HF_TOKEN"))
+]
+
+def get_tool(tool: str) -> GradioTool:


make this a class function

ntindle · 2023-04-26T06:14:30Z

src/autogpt_plugins/autogpt_gradio_tools/README.md

+1. Download this repository, and save it as `autogpt-gradio-tools.zip`
+2. Place the `.zip` file in the plugins directory of your AutoGPT install
+3. Add your twitter API information to the `.env` file within AutoGPT:


Remove this section

ntindle · 2023-04-26T06:16:00Z

src/autogpt_plugins/autogpt_gradio_tools/tools.py

+        hf_token=None,
+    ) -> None:
+        super().__init__(name, description, src, hf_token)
+        self.args = {"prompt": "text description of image"}


Got an error:

@ntindle ➜ /workspaces/Auto-GPT (master) $ AUTOGPT_GRADIO_TOOLS=StableDiffusion ./run.sh All packages are installed. NEWS: Welcome to Auto-GPT! We'll keep you informed of the latest news and features by printing messages here. If you don't wish to see this message, you can run Auto-GPT with the --skip-news flag # INCLUDED COMMAND 'send_tweet' IS DEPRICATED, AND WILL BE REMOVED IN THE NEXT STABLE RELEASE Base Twitter functionality (and more) is now covered by plugins: https://github.com/Significant-Gravitas/Auto-GPT-Plugins ## Changes to Docker configuration The workdir has been changed from /home/appuser to /app. Be sure to update any volume mounts accordingly. WARNING: You are running on `master` branch - this is not a supported branch. Loaded as API: https://gradio-client-demos-stable-diffusion.hf.space ✔ Loaded as API: https://taesiri-blip-2.hf.space ✔ Loaded as API: https://abidlabs-whisper.hf.space ✔ Loaded as API: https://damo-vilab-modelscope-text-to-video-synthesis.hf.space ✔ Loaded as API: https://microsoft-promptist.hf.space ✔ Loaded as API: https://abidlabs-docquery.hf.space ✔ Loaded as API: https://fffiloni-img-to-music.hf.space ✔ Loaded as API: https://pharma-clip-interrogator.hf.space ✔ WARNING: Plugin AutoGPTGradioTools found. But not in the allowlist... Load? (y/n): y WARNING: Plugin AutoGPTBingSearch found. But not in the allowlist... Load? (y/n): n WARNING: Plugin AutoGPTEmailPlugin found. But not in the allowlist... Load? (y/n): n WARNING: Plugin AutoGPTTwitter found. But not in the allowlist... Load? (y/n): n Plugins found: 1 -------------------- autogpt-gradio-tools: 0.1.0 - Calling Gradio Apps . Welcome back! Would you like me to return to being SearchGPT? Continue with the last settings? Name: SearchGPT Role: an AI assistant that specializes in conducting efficient and accurate online searches for users. Goals: ['Utilize advanced search algorithms and techniques to quickly and effectively find relevant information related to Taylor Swift.', 'Filter out irrelevant or low-quality search results to ensure that the user receives only the most useful and accurate information.', "Continuously learn and adapt to the user's search preferences and habits to provide a personalized and efficient search experience.", 'Provide additional context and information related to Taylor Swift, such as upcoming events, news articles, and social media updates.', 'Offer suggestions for related searches or topics that may be of interest to the user.'] API Budget: infinite Continue (y/n): n Welcome to Auto-GPT! run with '--help' for more information. Create an AI-Assistant: input '--manual' to enter manual mode. I want Auto-GPT to: Give me a bunch of generated images of taylor swift SwiftGPT has been created with the following details: Name: SwiftGPT Role: an AI image generator that specializes in creating high-quality, realistic images of Taylor Swift for fans and enthusiasts. Goals: - Generate a large number of high-quality images of Taylor Swift that are visually appealing and realistic. - Ensure that the generated images are diverse and capture different aspects of Taylor Swift's personality and style. - Continuously improve the quality of the generated images by incorporating feedback and using advanced image generation techniques. - Provide users with the ability to customize the generated images by adjusting various parameters such as pose, lighting, and background. - Ensure that the generated images are ethically and legally sound, and do not infringe on any copyrights or intellectual property rights. Using memory of type: LocalCache Using Browser: chrome THOUGHTS: I think I should start by generating some Taylor Swift images to fulfill my goals. I can use the 'stablediffusion' command to generate images based on text input. I will need to provide a text description of the image I want to generate. I will also need to save the generated images to a file for later use. REASONING: Generating Taylor Swift images is the main goal of this AI, and using the 'stablediffusion' command is the best way to achieve this. Saving the generated images to a file will allow me to use them later and improve the quality of the generated images. PLAN: - Use the 'stablediffusion' command to generate Taylor Swift images based on text input - Save the generated images to a file for later use CRITICISM: I need to ensure that the generated images are diverse and capture different aspects of Taylor Swift's personality and style. I also need to continuously improve the quality of the generated images by incorporating feedback and using advanced image generation techniques. NEXT ACTION: COMMAND = stablediffusion ARGUMENTS = {'prompt': 'Generate a Taylor Swift image based on a text description.'} Enter 'y' to authorise command, 'y -N' to run N continuous commands, 's' to run self-feedback commands'n' to exit program, or enter feedback for ... Input:y -=-=-=-=-=-=-= COMMAND AUTHORISED BY USER -=-=-=-=-=-=-= SYSTEM: Command stablediffusion returned: Error: GradioTool.run() got an unexpected keyword argument 'prompt' THOUGHTS: I think I should use the 'image_generator' command to generate Taylor Swift images based on text input. I will need to provide a text description of the image I want to generate. I will also need to save the generated images to a file for later use. REASONING: Generating Taylor Swift images is the main goal of this AI, and using the 'image_generator' command is the best way to achieve this. Saving the generated images to a file will allow me to use them later and improve the quality of the generated images. PLAN: - Use the 'image_generator' command to generate Taylor Swift images based on text input - Save the generated images to a file for later use CRITICISM: I need to ensure that the generated images are diverse and capture different aspects of Taylor Swift's personality and style. I also need to continuously improve the quality of the generated images by incorporating feedback and using advanced image generation techniques. NEXT ACTION: COMMAND = image_generator ARGUMENTS = {'prompt': 'Generate a Taylor Swift image based on a text description.'} Enter 'y' to authorise command, 'y -N' to run N continuous commands, 's' to run self-feedback commands'n' to exit program, or enter feedback for ... Input:

ntindle · 2023-04-26T06:21:14Z

src/autogpt_plugins/autogpt_gradio_tools/__init__.py

+        ts = [t for t in os.getenv("AUTOGPT_GRADIO_TOOLS", "").split(",") if t != ""]
+        self.tools = [get_tool(t) for t in ts]


Filter attempts to load this to names matching a tool. If you attempt to load an invalid name it will crash

@ntindle ➜ /workspaces/Auto-GPT (master) $ AUTOGPT_GRADIO_TOOLS=StableDiffusion,AutoGPTCaptioner ./run.sh All packages are installed. NEWS: Welcome to Auto-GPT! We'll keep you informed of the latest news and features by printing messages here. If you don't wish to see this message, you can run Auto-GPT with the --skip-news flag # INCLUDED COMMAND 'send_tweet' IS DEPRICATED, AND WILL BE REMOVED IN THE NEXT STABLE RELEASE Base Twitter functionality (and more) is now covered by plugins: https://github.com/Significant-Gravitas/Auto-GPT-Plugins ## Changes to Docker configuration The workdir has been changed from /home/appuser to /app. Be sure to update any volume mounts accordingly. WARNING: You are running on `master` branch - this is not a supported branch. Loaded as API: https://gradio-client-demos-stable-diffusion.hf.space ✔ Loaded as API: https://taesiri-blip-2.hf.space ✔ Loaded as API: https://abidlabs-whisper.hf.space ✔ Loaded as API: https://damo-vilab-modelscope-text-to-video-synthesis.hf.space ✔ Loaded as API: https://microsoft-promptist.hf.space ✔ Loaded as API: https://abidlabs-docquery.hf.space ✔ Loaded as API: https://fffiloni-img-to-music.hf.space ✔ Loaded as API: https://pharma-clip-interrogator.hf.space ✔ WARNING: Plugin AutoGPTGradioTools found. But not in the allowlist... Load? (y/n): y Traceback (most recent call last): File "/usr/local/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/local/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/workspaces/Auto-GPT/autogpt/__main__.py", line 5, in <module> autogpt.cli.main() File "/home/vscode/.local/lib/python3.10/site-packages/click/core.py", line 1130, in __call__ return self.main(*args, **kwargs) File "/home/vscode/.local/lib/python3.10/site-packages/click/core.py", line 1055, in main rv = self.invoke(ctx) File "/home/vscode/.local/lib/python3.10/site-packages/click/core.py", line 1635, in invoke rv = super().invoke(ctx) File "/home/vscode/.local/lib/python3.10/site-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File "/home/vscode/.local/lib/python3.10/site-packages/click/core.py", line 760, in invoke return __callback(*args, **kwargs) File "/home/vscode/.local/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func return f(get_current_context(), *args, **kwargs) File "/workspaces/Auto-GPT/autogpt/cli.py", line 90, in main run_auto_gpt( File "/workspaces/Auto-GPT/autogpt/main.py", line 104, in run_auto_gpt cfg.set_plugins(scan_plugins(cfg, cfg.debug_mode)) File "/workspaces/Auto-GPT/autogpt/plugins.py", line 229, in scan_plugins loaded_plugins.append(a_module()) File "/home/vscode/.local/lib/python3.10/site-packages/abstract_singleton/__init__.py", line 14, in __call__ cls._instances[cls] = super().__call__(*args, **kwargs) File "plugins/Auto-GPT-Plugins.zip/Auto-GPT-Plugins-add-gradio-tools/src/autogpt_plugins/autogpt_gradio_tools/__init__.py", line 55, in __init__ File "plugins/Auto-GPT-Plugins.zip/Auto-GPT-Plugins-add-gradio-tools/src/autogpt_plugins/autogpt_gradio_tools/__init__.py", line 55, in <listcomp> File "plugins/Auto-GPT-Plugins.zip/Auto-GPT-Plugins-add-gradio-tools/src/autogpt_plugins/autogpt_gradio_tools/__init__.py", line 36, in get_tool StopIteration Press any key to continue...

ntindle · 2023-04-26T06:23:42Z

src/autogpt_plugins/autogpt_gradio_tools/tools.py

+WORKSPACE_DIR = (Path(os.getcwd()) / "auto_gpt_workspace").resolve()
+
+
+class AutoGPTCaptioner(ImageCaptioningTool):


Match the function name to the tool name for all tools
i.e. ImageCaptioner

Error:

SYSTEM: Human feedback: Use the img /workspaces/Auto-GPT/autogpt/auto_gpt_workspace/taylor.jpeg THOUGHTS: I will attempt to caption the image taylor.jpeg REASONING: Now that I have verified that the image exists in my workspace, I can attempt to caption it PLAN: - Use the 'imagecaptioner' command to caption the image CRITICISM: I need to make sure the imagecaptioner command is installed and working properly NEXT ACTION: COMMAND = imagecaptioner ARGUMENTS = {'img': '/workspaces/Auto-GPT/autogpt/auto_gpt_workspace/taylor.jpeg'} Enter 'y' to authorise command, 'y -N' to run N continuous commands, 's' to run self-feedback commands'n' to exit program, or enter feedback for ... Input:y -=-=-=-=-=-=-= COMMAND AUTHORISED BY USER -=-=-=-=-=-=-= SYSTEM: Command imagecaptioner returned: Error: GradioTool.run() got an unexpected keyword argument 'img' THOUGHTS: I will verify if the image taylor.jpeg exists in my workspace REASONING: Before attempting to caption the image, I need to make sure it exists in my workspace PLAN: - Use the 'search_files' command to search for the image in my workspace CRITICISM: I need to make sure I am searching in the correct directory NEXT ACTION: COMMAND = search_files ARGUMENTS = {'directory': '/workspaces/Auto-GPT/autogpt/auto_gpt_workspace', 'filename': '/workspaces/Auto-GPT/autogpt/auto_gpt_workspace/taylor.jpeg'} Enter 'y' to authorise command, 'y -N' to run N continuous commands, 's' to run self-feedback commands'n' to exit program, or enter feedback for ... Input:

ntindle · 2023-04-26T06:24:50Z

src/autogpt_plugins/autogpt_gradio_tools/tools.py

+    def __init__(self, name="StableDiffusionPromptGenerator", description="Use this tool to improve a prompt for stable diffusion and other image and video generators. " "This tool will refine your prompt to include key words and phrases that make " "stable diffusion and other art generation algorithms perform better. The input is a prompt text string " "and the output is a prompt text string", src="microsoft/Promptist", hf_token=None) -> None:
+        super().__init__(name, description, src, hf_token)
+        self.args = {"prompt": "text description of image"}


Format this the same to the others

p-i- · 2023-05-06T01:25:33Z

This is a mass message from the AutoGPT core team to all open Plugin-PRs

We're in the process of re-architecting:
https://github.com/Significant-Gravitas/Auto-GPT/wiki/Architecting

Please be patient as we work to clean up the AutoGPT core.

If you already have an existing plug-in that’s affected, please message @ntindle on Discord so we can get your plugin spinning on the new
architecture.

If you're working on a plugin, please bear in mind that one of our primary goals in re-architecting is to make life MUCH easier for you.
We've taken aboard feedback from plugin developers. We're on it, so please give us some time!

<3 The AutoGPT Team

ntindle · 2023-05-10T02:29:20Z

@freddyaboulton is this something you plan on working on still?

lc0rp · 2023-05-19T13:38:52Z

Hi @freddyaboulton, we'd appreciate it if you could resolve conflicts so we can get this prepped for v0.3.2.

Just a heads up, the plugin interface may change soon. However, if you still plan on working on it, we're happy to get it in. If not, would you be okay with us forking it?

Thanks!

freddyaboulton · 2023-05-19T18:25:26Z

Sorry for the delay @lc0rp @ntindle ! I would love to get this in but I will have limited availability for the next week.

Feel free to fork and I can help review if you'd like to work on it in the meantime!

freddyaboulton · 2023-05-19T18:26:45Z

Sorry for the delay @lc0rp @ntindle ! I would love to get this in but I will have limited availability for the next week.

Feel free to fork and I can help review if you'd like to work on it in the meantime!

ntindle · 2023-05-30T15:38:18Z

No worries, pick it up when you can

freddyaboulton added 3 commits April 21, 2023 15:39

Add gradio-tools

9038116

Fixing

a8833ac

Fix typos

25cc465

freddyaboulton mentioned this pull request Apr 21, 2023

Extend abilities via gradio tools Significant-Gravitas/AutoGPT#1430

Closed

5 tasks

ntindle suggested changes Apr 22, 2023

View reviewed changes

freddyaboulton added 2 commits April 23, 2023 12:50

Merge branch 'master' of github.com:Significant-Gravitas/Auto-GPT-Plu…

def0335

…gins into add-gradio-tools

Add tests

1038ab1

freddyaboulton requested review from a team April 23, 2023 23:01

Fix typo

1fae35e

ntindle added the manual testing required label Apr 23, 2023

ntindle self-assigned this Apr 24, 2023

Merge branch 'master' into add-gradio-tools

0645a8e

ntindle suggested changes Apr 26, 2023

View reviewed changes

ntindle added this to the v0.3.1 release milestone May 2, 2023

ntindle modified the milestones: v0.3.1 release, v0.3.2 release May 14, 2023

lc0rp self-assigned this May 19, 2023

lc0rp added manual testing required and removed manual testing required labels May 19, 2023

lc0rp removed this from the v0.3.2 release milestone May 23, 2023

Merge branch 'master' into add-gradio-tools

271503a

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add gradio-tools plugin #14

Add gradio-tools plugin #14

freddyaboulton commented Apr 21, 2023 •

edited

Loading

ntindle commented Apr 21, 2023

ntindle Apr 22, 2023

ntindle Apr 22, 2023

ntindle Apr 22, 2023

freddyaboulton Apr 23, 2023

freddyaboulton commented Apr 23, 2023

ntindle commented Apr 23, 2023

ntindle left a comment

ntindle Apr 26, 2023

ntindle Apr 26, 2023

ntindle Apr 26, 2023

ntindle Apr 26, 2023

ntindle Apr 26, 2023

ntindle Apr 26, 2023

ntindle Apr 26, 2023

ntindle Apr 26, 2023

ntindle Apr 26, 2023

ntindle Apr 26, 2023

ntindle Apr 26, 2023

p-i- commented May 6, 2023

ntindle commented May 10, 2023

lc0rp commented May 19, 2023

freddyaboulton commented May 19, 2023

freddyaboulton commented May 19, 2023

ntindle commented May 30, 2023

		@@ -0,0 +1,239 @@
		"""Twitter API integrations using Tweepy."""

		import os


		WORKSPACE_DIR = (Path(os.getcwd()) / "auto_gpt_workspace").resolve()

		ts = [t for t in os.getenv("AUTOGPT_GRADIO_TOOLS", "").split(",") if t != ""]
		self.tools = [get_tool(t) for t in ts]

		WORKSPACE_DIR = (Path(os.getcwd()) / "auto_gpt_workspace").resolve()


		class AutoGPTCaptioner(ImageCaptioningTool):

Add gradio-tools plugin #14

Are you sure you want to change the base?

Add gradio-tools plugin #14

Conversation

freddyaboulton commented Apr 21, 2023 • edited Loading

ntindle commented Apr 21, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

freddyaboulton commented Apr 23, 2023

ntindle commented Apr 23, 2023

ntindle left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

p-i- commented May 6, 2023

ntindle commented May 10, 2023

lc0rp commented May 19, 2023

freddyaboulton commented May 19, 2023

freddyaboulton commented May 19, 2023

ntindle commented May 30, 2023

freddyaboulton commented Apr 21, 2023 •

edited

Loading