Datamigration from InfluxDB to QuestDB with Telegraf

Community
3

I have tried to migrate my data from influxdb2 using telegraf today. I have more than 1 mio values per month so i had to fiddle with limits and timeouts. Finally it worked by exporting the data per month using a simple script.

my telegraf config

[agent]
  omit_hostname = true

[[inputs.influxdb_v2_listener]]
  ## Address and port to host InfluxDB listener on
  ## (Double check the port. Needs to be available)
  service_address = ":8099"

[[aggregators.merge]]
 period = "876000h"  # Set the period to 100 years (876000 hours)
  grace = "876000h"  # Set the grace period to 100 years (876000 hours)
  drop_original = true
  metric_buffer_limit = 2000000

 [[outputs.influxdb_v2]]
  urls = ["http://127.0.0.1:9000"]
  metric_batch_size = 10000
  metric_buffer_limit = 2000000
   timeout = "60s"
  ping_timeout = "30s"
  flush_interval = "99999s"
  content_encoding = "identity"  # Important to ensuring no gzip encoding
~

and the script i used

!/bin/bash
# copy all data from influxdb2 to questdb using telegraf to transform data in rows
# we have to do it month by month as we have > 140000 values per month and telegraf
# has to perform the transformation in memory
#
# AD 10/2024
# !one time script, no error checking!

if [ $"$UID" != "0" ]; then
        echo "Influx export requires root"
        exit 1
fi

EP=/mpeg/data/influxdb-2/engine
BID=e1cffc49bb328fea
FN=influx_export

INV="\033[7m"
NOR="\033[0m"
BLD="\033[1m"

for YEAR in 2019 2020 2021 2022 2023 2024; do
        for MONTH in 01 02 03 04 05 06 07 08 09 10 11 12; do
                LASTDAY=$(cal --months=1 1 $MONTH $YEAR | awk 'NF {DAYS = $NF}; END {print DAYS}')
                START=$YEAR-$MONTH-01T00:00:00Z
                END=$YEAR-$MONTH-${LASTDAY}T23:59:59Z
                printf "\n${INV}Exporting $MONTH/$YEAR ($START to $END)${NOR}\n"
                influxd inspect export-lp --bucket-id $BID --engine-path $EP --output-path $FN --start $START --end $END 2>&1
                FILESIZE=$(stat -c%s "$FN")
                #echo "Size of $FN = $FILESIZE bytes."
                if [ $FILESIZE -le 1 ]; then
                        echo "No data for $MONTH/$YEAR"
                else
                        # start telegraf
                        telegraf --config sparse_to_dense.conf --debug &
                        TPID=$!
                        echo "Started telegraf with PID $TPID"
                        sleep 3
                        printf "${BLD}Sending data to telegraf${NORM}\n"
                        influx write --bucket ad --file $FN --host http://localhost:8099
                        printf "${BLD}waiting until telegraf has finished sending data to questdb, pid:${TPID}${NOR}\n"
                        kill $TPID
                        #killall telegraf
                        wait $TPID
                        printf "${BLD}-- end of processing for $MONTH/$YEAR --${NOR}\n"
                fi
        done
done
1 Like