Langtrace not logging costs on Vertex AI gemini-1.5-flash-002.

[arthurbrenno]

Hi. I'm using Vertex AI as my AI provider in my app. I'm logging the traces of the model calls successfully, but the model costs are still zero. Do I have to change any settings?

My dashboard:

Traces examples:

How I've setup langfuse:

langfuse_callback_handler = LlamaIndexCallbackHandler(
    public_key="...",
    secret_key="...",
    host="http://host.docker.internal:3000",
)
Settings.callback_manager = CallbackManager([langfuse_callback_handler])

How I'm using llamaindex (example):

from llama_index.llms.vertex import Vertex
from google.oauth2 import service_account
from llama_index.core.base.llms.types import ChatMessage, MessageRole
from llama_index.core.llms import LLM

credentials = service_account.Credentials.from_service_account_file("service-account.json")
llm: LLM = Vertex(model="gemini-1.5-flash-002", credentials=credentials)
response = await llm.achat(
    messages=[
             ChatMessage(role=MessageRole.SYSTEM, content="You are a helpful assistant."),
             ChatMessage(role=MessageRole.USER, content="Hi.")
    ]
)

[dosubot[bot]]

I found a similar discussion where the issue was that model costs were not showing up because Langfuse needs to ingest the cost data from the LLM response or infer it based on the model parameter ^[1].

In your case, Langtrace is not logging model costs on Vertex AI gemini-1.5-flash-002 because Langfuse either needs to ingest the cost data from the LLM response or infer it based on the model parameter. Since Langfuse does not have a predefined model definition for Vertex AI gemini-1.5-flash-002, you need to add a custom model definition for it ^[2].

Here’s how you can add a custom model definition:

1. Add Custom Model Definition:
   Ensure that the custom model definition includes the prices per unit (input, output, total) so that Langfuse can automatically calculate the cost for ingested generations.

   ## Custom model definitions
   You can flexibly add your own model definitions to Langfuse. This is especially useful for self-hosted or fine-tuned models which are not included in the list of Langfuse maintained models.
   
   Models are matched to generations based on:
   
   | Generation Attribute | Model Attribute | Notes |
   | --- | --- | --- |
   | `model` | `match_pattern` | Uses regular expressions, e.g. `(?i)^(gpt-4-0125-preview)$` matches `gpt-4-0125-preview`. |
   | `unit` | `unit` | Unit on the usage object of the generation (e.g. `TOKENS` or `CHARACTERS`) needs to match. |
   | `start_time` | `start_time` | Optional, can be used to update the price of a model without affecting past generations. If multiple models match, the model with the most recent `model.start_time` that is earlier than `generation.start_time` is used. |

2. Include Token Usage Data:
   Make sure to include token usage data when ingesting model generations. Langfuse cannot infer costs accurately without token usage data, especially for models that generate reasoning tokens.

By following these steps, you should be able to log the model costs correctly for Vertex AI gemini-1.5-flash-002 ^[3][4].

To continue talking to Dosu, mention @dosu.

[arthurbrenno]

Can someone help me on this? The bot answer seems to suggest a topic from an user that used langfuse without llamaindex. In my case, llamaindex, shoudn't it be registering the tokens and costs properlly?

[dosubot[bot]]

It looks like the issue you're experiencing with LlamaIndex not registering tokens and costs properly when using Langfuse with Vertex AI gemini-1.5-flash-002 is a known problem. There is an open issue on the Langfuse GitHub repository titled "bug: Instrumentation-based LlamaIndex integration (experimental) does not track model and costs correctly yet" ^[1]. Additionally, a support discussion on the Langfuse website titled "Langtrace not logging costs on Vertex AI gemini-1.5-flash-002" further corroborates this issue ^[2].

This seems to be a complex issue, and it might require more in-depth assistance. @marcklingen, could you help us out with this?

To continue talking to Dosu, mention @dosu.