"Google has deprecated this model on July 12, 2024, and the API request to https://generativelanguage.googleapis.com/v1beta/models/gemini-pro-vision:generateContent is returning a 404 Not Found status code."
Para solucionar, troquei o modelo para "gemini-1.5-flash" ou "gemini-1.5-pro-latest" e consegui contornar o erro que estava recebendo:
GoogleGenerativeAIFetchError: [GoogleGenerativeAI Error]: Error fetching from https://generativelanguage.googleapis.com/v1beta/models/gemini-pro-vision:generateContent: [404 Not Found] Gemini 1.0 Pro Vision has been deprecated on July 12, 2024. Consider switching to different model, for example gemini-1.5-flash.