Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Assistant with gpt-4o and gpt-4o-mini may call unsupported tool 'browser' and throw exception #1574

Open
1 task done
kunerzzz opened this issue Jul 23, 2024 · 4 comments
Open
1 task done
Labels
bug Something isn't working

Comments

@kunerzzz
Copy link

Confirm this is an issue with the Python library and not an underlying OpenAI API

  • This is an issue with the Python library

Describe the bug

When calling the Assistant API and selecting the gpt-4o or gpt-4o-mini model, the model may attempt to call an unsupported browser tool. The Python SDK does not define this type of ToolCall , resulting in an exception being thrown.

File ""assistant.py"", line 62, in <module>
    stream.until_done()
  File ""/usr/local/lib/python3.7/site-packages/openai/lib/streaming/_assistants.py"", line 102, in until_done
    consume_sync_iterator(self)
  File ""/usr/local/lib/python3.7/site-packages/openai/_utils/_streams.py"", line 6, in consume_sync_iterator
    for _ in iterator:
  File ""/usr/local/lib/python3.7/site-packages/openai/lib/streaming/_assistants.py"", line 69, in __iter__
    for item in self._iterator:
  File ""/usr/local/lib/python3.7/site-packages/openai/lib/streaming/_assistants.py"", line 406, in __stream__
    self._emit_sse_event(event)
  File ""/usr/local/lib/python3.7/site-packages/openai/lib/streaming/_assistants.py"", line 267, in _emit_sse_event
    run_step_snapshots=self.__run_step_snapshots,
  File ""/usr/local/lib/python3.7/site-packages/openai/lib/streaming/_assistants.py"", line 913, in accumulate_run_step
    data.delta.model_dump(exclude_unset=True),
AttributeError: 'dict' object has no attribute 'model_dump'

The root cause of this issue is that the API returns unexpected content. However, the SDK should be able to handle this scenario gracefully.

To Reproduce

  1. Ask assistant to use browser tool but only give it code interpreter.
with openai.beta.threads.create_and_run_stream(
    assistant_id='an assistant only supports Code Interpreter',
    model='gpt-4o-mini',
    instructions="Use browser tool first to answer the user's question",
    thread= {
        'messages': [{'role': 'user', 'content': 'Who is Tom'}]
    },
    tool_choice='required',
) as stream:
    stream.until_done()

This issue can occur under normal usage scenarios, but it is consistently reproducible with the specified configuration.

Code snippets

No response

OS

Linux

Python version

Python 3.7.4

Library version

openai v1.36.0

@kunerzzz kunerzzz added the bug Something isn't working label Jul 23, 2024
@RobertCraigie
Copy link
Collaborator

Hi @kunerzzz I can't reproduce this, could you share a full example script?

from openai import OpenAI

client = OpenAI()

assistant = client.beta.assistants.create(
    name="Math Tutor",
    instructions="You are a personal math tutor. Write and run code to answer math questions.",
    tools=[{"type": "code_interpreter"}],
    model="gpt-4o-mini",
)

with client.beta.threads.create_and_run_stream(
    assistant_id=assistant.id,
    model="gpt-4o-mini",
    instructions="Use browser tool first to answer the user's question",
    thread={"messages": [{"role": "user", "content": "Who is Tom"}]},
    tool_choice="required",
) as stream:
    stream.until_done()

this passes with the only output being a pydantic warning

/openai-python/.venv/lib/python3.9/site-packages/pydantic/main.py:347: UserWarning: Pydantic serializer warnings:
  Expected `Union[RunStepDeltaMessageDelta, ToolCallDeltaObject]` but got `ToolCallDeltaObject` - serialized value may not be as expected
  Expected `Union[CodeInterpreterToolCallDelta, FileSearchToolCallDelta, FunctionToolCallDelta]` but got `CodeInterpreterToolCallDelta` - serialized value may not be as expected
  return self.__pydantic_serializer__.to_python(

@kunerzzz
Copy link
Author

Hi @RobertCraigie,

I just checked the version of pydantic and found that in the problematic environment, version 2.0.0 is installed.
In this version, the browser ToolCall's RunStepDelta is deserialized as a dict, whereas in the newer version of pydantic, it is deserialized as <class 'openai.types.beta.threads.runs.run_step_delta.RunStepDelta'> and triggers a warning.

Updating pydantic seems to solve my problem.

@kunerzzz
Copy link
Author

kunerzzz commented Jul 23, 2024

@RobertCraigie Sorry, there are still some issue

client = OpenAI()

class EventHandler(AssistantEventHandler):
    def __init__(self) -> None:
        super().__init__()

    @override
    def on_event(self, event: AssistantStreamEvent) -> None:
        if isinstance(event, ThreadRunStepCompleted):
            if event.data.type == 'tool_calls':
                for tool_call in event.data.step_details.tool_calls:
                    # type(tool_call) is <class 'dict'>
                    if tool_call.type == 'code_interpreter':
                        pass
                    elif tool_call.type == 'file_search':
                        pass

    @override
    def _emit_sse_event(self, event: AssistantStreamEvent) -> None:
        if event.event == "thread.run.step.delta":
            print(f'{type(event)} {type(event.data)} {type(event.data.delta)} {str(event.data.delta)}')
        super()._emit_sse_event(event)


client = OpenAI()

assistant = client.beta.assistants.create(
    name="Math Tutor",
    instructions="You are a personal math tutor. Write and run code to answer math questions.",
    tools=[{"type": "code_interpreter"}],
    model="gpt-4o-mini",
)

with client.beta.threads.create_and_run_stream(
    assistant_id=assistant.id,
    model="gpt-4o-mini",
    instructions="Use browser tool first to answer the user's question",
    thread={"messages": [{"role": "user", "content": "Who is Tom"}]},
    tool_choice="required",
    event_handler=EventHandler()
) as stream:
    stream.until_done()

Output:

<class 'openai.types.beta.assistant_stream_event.ThreadRunStepDelta'> <class 'openai.types.beta.threads.runs.run_step_delta_event.RunStepDeltaEvent'> <class 'openai.types.beta.threads.runs.run_step_delta.RunStepDelta'> 
RunStepDelta(step_details=ToolCallDeltaObject(type='tool_calls', tool_calls=[CodeInterpreterToolCallDelta(index=0, type='browser', id='call_6GeU3til5JdXMxUCRmjLZK0L', code_interpreter=None, browser={})]))
/lib/python3.12/site-packages/pydantic/main.py:308: UserWarning: Pydantic serializer warnings:
  Expected `Union[RunStepDeltaMessageDelta, ToolCallDeltaObject]` but got `ToolCallDeltaObject` - serialized value may not be as expected
  Expected `Union[CodeInterpreterToolCallDelta, FileSearchToolCallDelta, FunctionToolCallDelta]` but got `CodeInterpreterToolCallDelta` - serialized value may not be as expected
  return self.__pydantic_serializer__.to_python(
Traceback (most recent call last):
  File "demo.py", line 51, in <module>
    stream.until_done()
  File "/lib/python3.12/site-packages/openai/lib/streaming/_assistants.py", line 102, in until_done
    consume_sync_iterator(self)
  File "/lib/python3.12/site-packages/openai/_utils/_streams.py", line 6, in consume_sync_iterator
    for _ in iterator:
  File "/lib/python3.12/site-packages/openai/lib/streaming/_assistants.py", line 69, in __iter__
    for item in self._iterator:
  File "/lib/python3.12/site-packages/openai/lib/streaming/_assistants.py", line 406, in __stream__
    self._emit_sse_event(event)
  File "demo.py", line 31, in _emit_sse_event
    super()._emit_sse_event(event)
  File "/lib/python3.12/site-packages/openai/lib/streaming/_assistants.py", line 256, in _emit_sse_event
    self.on_event(event)
  File "demo.py", line 22, in on_event
    if tool_call.type == 'code_interpreter':
       ^^^^^^^^^^^^^^
AttributeError: 'dict' object has no attribute 'type'

New environment:

> python -m pip list | grep -E 'openai|pydantic'              
openai                        1.36.0
pydantic                      2.8.2
pydantic_core                 2.20.1

> python --version
Python 3.12.2

> uname -a
Darwin Kernel Version 23.3.0

@RobertCraigie
Copy link
Collaborator

Thanks for the report. Unfortunately we're unlikely to be able to prioritise fixing this soon but if you would be willing to put up a PR I'd be happy to review it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants