Не удается создать журналы при запуске потребителя kafka после установки cdc на сервере sql 2019 и регистрации соединителя сервера debezium sql

Ранее я уже настраивал CDC в SQL Server на основе этого ссылка и зарегистрируйте соединитель Debezium SQL Server для подключения Kafka на основе этого ссылка. Я запускаю сценарий потребителя Kafka и делаю некоторые данные модификации (вставка, обновление и удаление) в SQL Server, но потребитель Kafka не отображает никаких журналов JSON о данных модификации.

Это мой файл конфигурации sqlserver-debezium.json:

{
   "name":"sqlserver-debezium",
   "config":{
      "connector.class":"io.debezium.connector.sqlserver.SqlServerConnector",
      "database.user":"sa",
      "database.dbname":"cdc_sqlserver",
      "database.history.kafka.bootstrap.servers":"10.140.0.33:9092",
      "database.history.kafka.topic":"debezium_sqlserver",
      "database.server.name":"sqlserver-debezium",
      "database.port":"1433",
      "table.whitelist":"cdc_sqlserver_ipul.dbo.customers",
      "database.hostname":"10.140.15.214",
      "database.password":"test123",
      "name":"sqlserver-debezium",
      "include.query":"true",
      "snapshot.mode":"when_needed"
   }
}

Я добавляю коннектор через хост-машину следующим образом (как в руководстве):

curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" localhost:8083/connectors/ -d @sqlserver-debezium.json

Запускаем скрипт потребителя Kafka:

kafka-console-consumer --bootstrap-server localhost:9092 --topic sqlserver-connector --from-beginning

Буду признателен за любую помощь.


apache-kafka sql-server apache-kafka-connect debezium
person secretCode    schedule 04.05.2020    source источник
comment
Добро пожаловать в StackOverflow! Можете ли вы проверить журналы работника Kafka Connect на наличие ошибок и обновить свой вопрос, чтобы включить их?   -  person Robin Moffatt    schedule 05.05.2020

Ответы (2)


arrow_upward
0
arrow_downward

Не могли бы вы дважды проверить настройку table.whitelist? Должно быть только dbo.customers.

person Jiri Pechanec    schedule 05.05.2020
comment
Я уже меняю значение table.whitelist становится dbo.customers, но все еще не отображает файл журналов JSON в потребителе kafka - person secretCode; 06.05.2020

arrow_upward
0
arrow_downward

Эта проблема решена. Итак, я изменяю значение переменной snapshot.mode на initial, удаляю variabel include.query в sqlserver-debezium.json, повторяю попытку соединителя регистра и делаю некоторые изменения (вставка, обновление, удаление) в базе данных сервера sql, затем отображаю журналы json, подобные этому

{"schema":{"type":"struct","fields":[{"type":"struct","fields":[{"type":"int32","optional":false,"field":"id"},{"type":"string","optional":true,"field":"first_name"
},{"type":"string","optional":true,"field":"last_name"},{"type":"string","optional":true,"field":"email"}],"optional":true,"name":"btpns_sqlserver_ipul.dbo.customer
s.Value","field":"before"},{"type":"struct","fields":[{"type":"int32","optional":false,"field":"id"},{"type":"string","optional":true,"field":"first_name"},{"type":
"string","optional":true,"field":"last_name"},{"type":"string","optional":true,"field":"email"}],"optional":true,"name":"btpns_sqlserver_ipul.dbo.customers.Value","
field":"after"},{"type":"struct","fields":[{"type":"string","optional":false,"field":"version"},{"type":"string","optional":false,"field":"connector"},{"type":"stri
ng","optional":false,"field":"name"},{"type":"int64","optional":false,"field":"ts_ms"},{"type":"string","optional":true,"name":"io.debezium.data.Enum","version":1,"
parameters":{"allowed":"true,last,false"},"default":"false","field":"snapshot"},{"type":"string","optional":false,"field":"db"},{"type":"string","optional":false,"f
ield":"schema"},{"type":"string","optional":false,"field":"table"},{"type":"string","optional":true,"field":"change_lsn"},{"type":"string","optional":true,"field":"
commit_lsn"},{"type":"int64","optional":true,"field":"event_serial_no"}],"optional":false,"name":"io.debezium.connector.sqlserver.Source","field":"source"},{"type":
"string","optional":false,"field":"op"},{"type":"int64","optional":true,"field":"ts_ms"},{"type":"struct","fields":[{"type":"string","optional":false,"field":"id"},
{"type":"int64","optional":false,"field":"total_order"},{"type":"int64","optional":false,"field":"data_collection_order"}],"optional":true,"field":"transaction"}],"
optional":false,"name":"btpns_sqlserver_ipul.dbo.customers.Envelope"},"payload":{"before":{"id":9,"first_name":"Andi","last_name":"Thompson","email":"[email protected]"
},"after":{"id":9,"first_name":"Deni","last_name":"Thompson","email":"[email protected]"},"source":{"version":"1.2.0-SNAPSHOT","connector":"sqlserver","name":"btpns-sql
server-ipul","ts_ms":1588740858363,"snapshot":"false","db":"btpns_cdc_sqlserver_ipul","schema":"dbo","table":"customers","change_lsn":"0000002d:00003da0:0002","comm
it_lsn":"0000002d:00003da0:0003","event_serial_no":2},"op":"u","ts_ms":1588740859225,"transaction":null}}
{"schema":{"type":"struct","fields":[{"type":"struct","fields":[{"type":"int32","optional":false,"field":"id"},{"type":"string","optional":true,"field":"first_name"
},{"type":"string","optional":true,"field":"last_name"},{"type":"string","optional":true,"field":"email"}],"optional":true,"name":"btpns_sqlserver_ipul.dbo.customer
s.Value","field":"before"},{"type":"struct","fields":[{"type":"int32","optional":false,"field":"id"},{"type":"string","optional":true,"field":"first_name"},{"type":
"string","optional":true,"field":"last_name"},{"type":"string","optional":true,"field":"email"}],"optional":true,"name":"btpns_sqlserver_ipul.dbo.customers.Value","
field":"after"},{"type":"struct","fields":[{"type":"string","optional":false,"field":"version"},{"type":"string","optional":false,"field":"connector"},{"type":"stri
ng","optional":false,"field":"name"},{"type":"int64","optional":false,"field":"ts_ms"},{"type":"string","optional":true,"name":"io.debezium.data.Enum","version":1,"
parameters":{"allowed":"true,last,false"},"default":"false","field":"snapshot"},{"type":"string","optional":false,"field":"db"},{"type":"string","optional":false,"f
ield":"schema"},{"type":"string","optional":false,"field":"table"},{"type":"string","optional":true,"field":"change_lsn"},{"type":"string","optional":true,"field":"
commit_lsn"},{"type":"int64","optional":true,"field":"event_serial_no"}],"optional":false,"name":"io.debezium.connector.sqlserver.Source","field":"source"},{"type":
"string","optional":false,"field":"op"},{"type":"int64","optional":true,"field":"ts_ms"},{"type":"struct","fields":[{"type":"string","optional":false,"field":"id"},
{"type":"int64","optional":false,"field":"total_order"},{"type":"int64","optional":false,"field":"data_collection_order"}],"optional":true,"field":"transaction"}],"
optional":false,"name":"btpns_sqlserver_ipul.dbo.customers.Envelope"},"payload":{"before":{"id":9,"first_name":"Deni","last_name":"Thompson","email":"[email protected]"
},"after":null,"source":{"version":"1.2.0-SNAPSHOT","connector":"sqlserver","name":"btpns-sqlserver-ipul","ts_ms":1588744428530,"snapshot":"false","db":"btpns_cdc_s
qlserver_ipul","schema":"dbo","table":"customers","change_lsn":"0000002d:00005400:0002","commit_lsn":"0000002d:00005400:0005","event_serial_no":1},"op":"d","ts_ms":
1588744430725,"transaction":null}}
person secretCode    schedule 06.05.2020