Entrar Ainda não tem acesso?
Solucionado (ver solução)
Solucionado
(ver solução)
2
respostas

[Dúvida] Falha ao se conectar aos serviços do Kafka

Referente ao curso Kafka: produtores, consumidores e streams, no capítulo Produtores e consumidores e atividade Faça como eu fiz na aula

por Marcos Sousa
| 116k xp | 37 posts
ACP - Engenharia de Software

Criei este seguinte docker-compose para subir os serviços do zookeeper e do kafka

version: '2'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.3.5
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    ports:
      - 22181:2181
  
  kafka:
    image: confluentinc/cp-kafka:7.3.5
    depends_on:
      - zookeeper
    ports:
      - 29092:29092
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

O serviço e startado sem erros

        Name                     Command            State                               Ports                             
--------------------------------------------------------------------------------------------------------------------------
ecommerce_kafka_1       /etc/confluent/docker/run   Up      0.0.0.0:29092->29092/tcp,:::29092->29092/tcp, 9092/tcp        
ecommerce_zookeeper_1   /etc/confluent/docker/run   Up      0.0.0.0:22181->2181/tcp,:::22181->2181/tcp, 2888/tcp, 3888/tcp

Quando executo o cliente tenho o seguinte erro

[main] WARN org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=FraudeDetectorService] Connection to node -1 (/127.0.0.1:9092) could not be established. Broker may not be available.
[main] WARN org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=FraudeDetectorService] Connection to node -1 (/127.0.0.1:9092) could not be established. Broker may not be available.

O mesmo acontece com o Produtor

[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka version: 2.3.1
[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka commitId: 18a913733fb71c01
[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka startTimeMs: 1696506533062
[kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node -1 (/127.0.0.1:9092) could not be established. Broker may not be available.
[kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node -1 (/127.0.0.1:9092) could not be established. Broker may not be available.

Como posso resolver este erro?

2 respostas
solução!
por Marcos Sousa
| 116k xp | 37 posts
ACP - Engenharia de Software

Fiz o seguinte ajuste no docker-compose

  kafka:
    image: confluentinc/cp-kafka:7.3.5
    depends_on:
      - zookeeper
    ports:
      - 9092:9092
    environment:
      KAFKA_ADVERTISED_HOST_NAME: localhost
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      //Troca de kafka:9092 para kafka:29092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

Ao executar o producer tive a seguinte saída

sl.truststore.password = null
    ssl.truststore.type = JKS
    transaction.timeout.ms = 60000
    transactional.id = null
    value.serializer = class org.apache.kafka.common.serialization.StringSerializer

[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka version: 2.3.1
[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka commitId: 18a913733fb71c01
[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka startTimeMs: 1696509208241
[kafka-producer-network-thread | producer-1] INFO org.apache.kafka.clients.Metadata - [Producer clientId=producer-1] Cluster ID: x1ND0awSRcusz54K-8r4Mw

Ao rodar o consumer tive a seguinte saída

 INFO org.apache.kafka.common.utils.AppInfoParser - Kafka commitId: 18a913733fb71c01
[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka startTimeMs: 1696509332550
[main] INFO org.apache.kafka.clients.consumer.KafkaConsumer - [Consumer clientId=consumer-1, groupId=FraudeDetectorService] Subscribed to topic(s): ECOMMERCE_NEW_ORDER
[main] INFO org.apache.kafka.clients.Metadata - [Consumer clientId=consumer-1, groupId=FraudeDetectorService] Cluster ID: x1ND0awSRcusz54K-8r4Mw
[main] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=FraudeDetectorService] Discovered group coordinator localhost:9092 (id: 2147483646 rack: null)
[main] INFO org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=FraudeDetectorService] Revoking previously assigned partitions []
[main] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=FraudeDetectorService] (Re-)joining group
[main] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=FraudeDetectorService] (Re-)joining group
[main] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=FraudeDetectorService] Successfully joined group with generation 4
[main] INFO org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=FraudeDetectorService] Setting newly assigned partitions: ECOMMERCE_NEW_ORDER-0
[main] INFO org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=FraudeDetectorService] Setting offset for partition ECOMMERCE_NEW_ORDER-0 to the committed offset FetchPosition{offset=0, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=localhost:9092 (id: 1 rack: null), epoch=0}}

Em principio o erro pode estar no producer. Instalei este cliente para listar os tópicos do Kafka https://kafkatool.com/download.html e quando enviei um valor por ele o consumer processou a mensagem.

Insira aqui a descrição dessa imagem para ajudar na acessibilidade

Resultado da mensagem processada.

por Marcos Sousa
| 116k xp | 37 posts
ACP - Engenharia de Software

Segue o link do meu repositório caso queiram dar uma olhada

https://github.com/estevamdf2/ecommerce