Data is not being saved using the nodejs package

Community
1

I’m experiencing an issue, but I’m not sure what’s happening.

I have a Node.js script that retrieves GPS data from an API and inserts it into QuestDB. This script runs as a cron job every 10 seconds.

The first time the cron job runs, the data is saved correctly. However, in subsequent runs, some values are missing. For example, the serverTimestamp, lat, and lon columns are not being populated. Strangely, the timestamp, carro, and linha values are correct.

Environment:

  • Node.js version: 18.18.1
  • QuestDB: Docker image 8.0.1, I’m using the default configuration

The table schema is like this.

CREATE TABLE proj_5_raw_positions (
    timestamp TIMESTAMP,
    serverTimestamp TIMESTAMP,
    carro SYMBOL,
    linha SYMBOL,
    lat DOUBLE,
    lon DOUBLE,
    bearing INT
) TIMESTAMP(timestamp) PARTITION BY DAY WAL
DEDUP UPSERT KEYS(timestamp, carro);

I made this sample script and some sample data.

const { Sender } = require('@questdb/nodejs-client');
const { addHours, parse } = require('date-fns');
const allCycleData = require('./all-cycle-data.js');

async function saveData(data) {
  const startedAt = new Date();
  console.log('-------------------------');
  console.log('Starting at: ', startedAt.toISOString());
  const sender = Sender.fromConfig('http::addr=localhost:9000');

  for (const pos of Object.values(data)) {
    const rawPosition = {
      timestamp: addHours(
        parse(pos.dataHora, 'dd/MM/yyyy HH:mm:ss', new Date()),
        3,
      ).getTime(),
      serverTimestamp: startedAt.getTime(),
      carro: pos.carro,
      lat: Number(pos.latitude),
      lon: Number(pos.longitude),
      linha: pos.linha,
      bearing: 0,
    };
    console.log(rawPosition);

    await sender
      .table('proj_5_raw_positions')
      .symbol('carro', rawPosition.carro)
      .symbol('linha', rawPosition.linha)
      .timestampColumn('serverTimestamp', rawPosition.serverTimestamp, 'ms')
      .floatColumn('lat', rawPosition.lat)
      .floatColumn('lon', rawPosition.lon)
      .intColumn('bearing', rawPosition.bearing)
      .at(rawPosition.timestamp, 'ms');
    // INFO: With this flush here the data takes longer to appear on the webui query
    // and has more missing fields value
    // await sender.flush();
  }

  // INFO: I made this do while because calling flush only one time is not flushing all data
  let stillHasDataInBuffer = false;
  do {
    stillHasDataInBuffer = await sender.flush();
    console.log('stillHasDataInBuffer: ', stillHasDataInBuffer);
  } while (stillHasDataInBuffer);

  await sender.close();

  console.log('Finished insert at: ', new Date().toISOString());
}

(async () => {
  for (const cycleData of allCycleData) {
    await saveData(cycleData);
    await new Promise((resolve) => setTimeout(resolve, 10000));
  }
})();

Sample data all-cycle-data.js

module.exports = [
  {
    9102: {
      carro: '9102',
      linha: '10',
      dataHora: '11/07/2024 07:45:35',
      latitude: '-26.91596500',
      longitude: '-49.07206300',
    },
    9103: {
      carro: '9103',
      linha: '10',
      dataHora: '11/07/2024 07:45:40',
      latitude: '-26.94637200',
      longitude: '-49.07230500',
    },
    9110: {
      carro: '9110',
      linha: '10',
      dataHora: '11/07/2024 07:45:31',
      latitude: '-26.87318900',
      longitude: '-49.11418700',
    },
    9115: {
      carro: '9115',
      linha: '10',
      dataHora: '11/07/2024 07:45:34',
      latitude: '-26.91122600',
      longitude: '-49.07068300',
    },
    9120: {
      carro: '9120',
      linha: '10',
      dataHora: '11/07/2024 07:45:45',
      latitude: '-26.87803800',
      longitude: '-49.08680400',
    },
    9124: {
      carro: '9124',
      linha: '607',
      dataHora: '11/07/2024 07:45:42',
      latitude: '-26.92347600',
      longitude: '-49.06223000',
    },
    9129: {
      carro: '9129',
      linha: '15',
      dataHora: '11/07/2024 07:45:43',
      latitude: '-26.92827000',
      longitude: '-49.05740700',
    },
    9132: {
      carro: '9132',
      linha: '30',
      dataHora: '11/07/2024 07:45:43',
      latitude: '-26.92785400',
      longitude: '-49.11353900',
    },
    9134: {
      carro: '9134',
      linha: '300',
      dataHora: '11/07/2024 07:45:32',
      latitude: '-26.91807100',
      longitude: '-49.13453200',
    },
    9138: {
      carro: '9138',
      linha: '403',
      dataHora: '11/07/2024 07:44:47',
      latitude: '-26.99609117',
      longitude: '-49.10035847',
    },
    9140: {
      carro: '9140',
      linha: '11',
      dataHora: '11/07/2024 07:44:57',
      latitude: '-26.87066847',
      longitude: '-49.09575328',
    },
    9144: {
      carro: '9144',
      linha: '153',
      dataHora: '11/07/2024 07:45:42',
      latitude: '-26.87707300',
      longitude: '-49.12943100',
    },
    9149: {
      carro: '9149',
      linha: '15',
      dataHora: '11/07/2024 07:45:34',
      latitude: '-26.89804600',
      longitude: '-49.07637900',
    },
    9152: {
      carro: '9152',
      linha: '31',
      dataHora: '11/07/2024 07:45:33',
      latitude: '-26.91506700',
      longitude: '-49.10928500',
    },
    9159: {
      carro: '9159',
      linha: '124',
      dataHora: '11/07/2024 07:45:42',
      latitude: '-26.85415500',
      longitude: '-49.13115100',
    },
    9161: {
      carro: '9161',
      linha: '32',
      dataHora: '11/07/2024 07:45:41',
      latitude: '-26.92042500',
      longitude: '-49.06377400',
    },
    9171: {
      carro: '9171',
      linha: '32',
      dataHora: '11/07/2024 07:45:43',
      latitude: '-26.90985400',
      longitude: '-49.12067800',
    },
    9172: {
      carro: '9172',
      linha: '123',
      dataHora: '11/07/2024 07:45:40',
      latitude: '-26.85370200',
      longitude: '-49.12942100',
    },
    9173: {
      carro: '9173',
      linha: '31',
      dataHora: '11/07/2024 07:45:37',
      latitude: '-26.92826400',
      longitude: '-49.05780800',
    },
    9188: {
      carro: '9188',
      linha: '12',
      dataHora: '11/07/2024 07:45:31',
      latitude: '-26.91258200',
      longitude: '-49.07971500',
    },
    9190: {
      carro: '9190',
      linha: '32',
      dataHora: '11/07/2024 07:45:36',
      latitude: '-26.93433900',
      longitude: '-49.13034100',
    },
    9193: {
      carro: '9193',
      linha: '12',
      dataHora: '11/07/2024 07:45:11',
      latitude: '-26.87087065',
      longitude: '-49.09498114',
    },
    9194: {
      carro: '9194',
      linha: '30',
      dataHora: '11/07/2024 07:45:28',
      latitude: '-26.91279900',
      longitude: '-49.08096300',
    },
    9195: {
      carro: '9195',
      linha: '31',
      dataHora: '11/07/2024 07:45:43',
      latitude: '-26.92308400',
      longitude: '-49.06323000',
    },
    9196: {
      carro: '9196',
      linha: '12',
      dataHora: '11/07/2024 07:45:40',
      latitude: '-26.91828600',
      longitude: '-49.07018600',
    },
    9304: {
      carro: '9304',
      linha: '603',
      dataHora: '11/07/2024 07:45:35',
      latitude: '-26.87560500',
      longitude: '-49.05584100',
    },
    9310: {
      carro: '9310',
      linha: '302',
      dataHora: '11/07/2024 07:45:36',
      latitude: '-26.93724600',
      longitude: '-49.14896000',
    },
    9311: {
      carro: '9311',
      linha: '32',
      dataHora: '11/07/2024 07:45:49',
      latitude: '-26.91407900',
      longitude: '-49.08372100',
    },
    9446: {
      carro: '9446',
      linha: '81',
      dataHora: '11/07/2024 07:27:27',
      latitude: '-26.79860045',
      longitude: '-49.08510827',
    },
    9448: {
      carro: '9448',
      linha: '504',
      dataHora: '11/07/2024 07:45:36',
      latitude: '-26.92934500',
      longitude: '-49.05445700',
    },
    9449: {
      carro: '9449',
      linha: '601',
      dataHora: '11/07/2024 07:45:15',
      latitude: '-26.84884600',
      longitude: '-49.04986300',
    },
    9451: {
      carro: '9451',
      linha: '300',
      dataHora: '11/07/2024 07:45:35',
      latitude: '-26.93397700',
      longitude: '-49.13050700',
    },
    9452: {
      carro: '9452',
      linha: '125',
      dataHora: '11/07/2024 07:45:25',
      latitude: '-26.87061300',
      longitude: '-49.09536600',
    },
    9453: {
      carro: '9453',
      linha: '81',
      dataHora: '11/07/2024 07:45:26',
      latitude: '-26.82613400',
      longitude: '-49.08698700',
    },
    9461: {
      carro: '9461',
      linha: '804',
      dataHora: '11/07/2024 07:45:28',
      latitude: '-26.72177300',
      longitude: '-49.06078700',
    },
    9464: {
      carro: '9464',
      linha: '616',
      dataHora: '11/07/2024 07:45:38',
      latitude: '-26.88322900',
      longitude: '-49.08483700',
    },
    9468: {
      carro: '9468',
      linha: '111',
      dataHora: '11/07/2024 07:45:42',
      latitude: '-26.87063000',
      longitude: '-49.09506600',
    },
    9470: {
      carro: '9470',
      linha: '154',
      dataHora: '11/07/2024 07:45:25',
      latitude: '-26.88218200',
      longitude: '-49.14017400',
    },
    9476: {
      carro: '9476',
      linha: '80',
      dataHora: '11/07/2024 07:45:32',
      latitude: '-26.86222300',
      longitude: '-49.08215500',
    },
    9478: {
      carro: '9478',
      linha: '904',
      dataHora: '11/07/2024 07:45:38',
      latitude: '-26.90168700',
      longitude: '-49.09874100',
    },
    9480: {
      carro: '9480',
      linha: '81',
      dataHora: '11/07/2024 07:45:44',
      latitude: '-26.86681800',
      longitude: '-49.08738200',
    },
    9482: {
      carro: '9482',
      linha: '122',
      dataHora: '11/07/2024 07:45:44',
      latitude: '-26.79351800',
      longitude: '-49.08716700',
    },
    9485: {
      carro: '9485',
      linha: '120',
      dataHora: '11/07/2024 07:45:28',
      latitude: '-26.87115000',
      longitude: '-49.09480200',
    },
    9487: {
      carro: '9487',
      linha: '508',
      dataHora: '11/07/2024 07:45:42',
      latitude: '-26.91568100',
      longitude: '-49.06221000',
    },
    9490: {
      carro: '9490',
      linha: '402',
      dataHora: '11/07/2024 07:45:31',
      latitude: '-26.96593300',
      longitude: '-49.07340000',
    },
    9491: {
      carro: '9491',
      linha: '507',
      dataHora: '11/07/2024 07:45:25',
      latitude: '-26.90881326',
      longitude: '-49.02324932',
    },
    9492: {
      carro: '9492',
      linha: '405',
      dataHora: '11/07/2024 07:33:35',
      latitude: '-26.99532095',
      longitude: '-49.07100898',
    },
    9493: {
      carro: '9493',
      linha: '705',
      dataHora: '11/07/2024 07:45:26',
      latitude: '-26.90864500',
      longitude: '-49.07638500',
    },
    9497: {
      carro: '9497',
      linha: '605',
      dataHora: '11/07/2024 07:45:37',
      latitude: '-26.89549200',
      longitude: '-49.07089900',
    },
    9500: {
      carro: '9500',
      linha: '308',
      dataHora: '11/07/2024 07:45:39',
      latitude: '-26.93388300',
      longitude: '-49.13071300',
    },
    9502: {
      carro: '9502',
      linha: '704',
      dataHora: '11/07/2024 07:45:32',
      latitude: '-26.91368300',
      longitude: '-49.14031500',
    },
    9503: {
      carro: '9503',
      linha: '606',
      dataHora: '11/07/2024 07:45:37',
      latitude: '-26.86977500',
      longitude: '-49.07313500',
    },
    9706: {
      carro: '9706',
      linha: '423',
      dataHora: '11/07/2024 07:45:21',
      latitude: '-26.97433527',
      longitude: '-49.05523164',
    },
    9713: {
      carro: '9713',
      linha: '305',
      dataHora: '11/07/2024 07:45:40',
      latitude: '-26.95045800',
      longitude: '-49.14717900',
    },
    9717: {
      carro: '9717',
      linha: '509',
      dataHora: '11/07/2024 07:45:35',
      latitude: '-26.95341300',
      longitude: '-49.08201400',
    },
  },
  {
    9102: {
      carro: '9102',
      linha: '10',
      dataHora: '11/07/2024 07:55:34',
      latitude: '-26.93136600',
      longitude: '-49.05961100',
    },
    9103: {
      carro: '9103',
      linha: '10',
      dataHora: '11/07/2024 07:55:38',
      latitude: '-26.92198600',
      longitude: '-49.05758800',
    },
    9115: {
      carro: '9115',
      linha: '10',
      dataHora: '11/07/2024 07:55:34',
      latitude: '-26.88403800',
      longitude: '-49.08439600',
    },
    9118: {
      carro: '9118',
      linha: '10',
      dataHora: '11/07/2024 07:55:44',
      latitude: '-26.87086500',
      longitude: '-49.09519400',
    },
    9120: {
      carro: '9120',
      linha: '10',
      dataHora: '11/07/2024 07:55:24',
      latitude: '-26.90603400',
      longitude: '-49.07582600',
    },
    9123: {
      carro: '9123',
      linha: '405',
      dataHora: '11/07/2024 07:55:45',
      latitude: '-26.96235200',
      longitude: '-49.06553500',
    },
    9129: {
      carro: '9129',
      linha: '15',
      dataHora: '11/07/2024 07:55:42',
      latitude: '-26.92014400',
      longitude: '-49.06410000',
    },
    9132: {
      carro: '9132',
      linha: '31',
      dataHora: '11/07/2024 07:55:42',
      latitude: '-26.92917500',
      longitude: '-49.12262100',
    },
    9134: {
      carro: '9134',
      linha: '300',
      dataHora: '11/07/2024 07:55:32',
      latitude: '-26.89470000',
      longitude: '-49.13228000',
    },
    9138: {
      carro: '9138',
      linha: '403',
      dataHora: '11/07/2024 07:55:01',
      latitude: '-26.97057095',
      longitude: '-49.07527462',
    },
    9140: {
      carro: '9140',
      linha: '11',
      dataHora: '11/07/2024 07:55:27',
      latitude: '-26.88125859',
      longitude: '-49.08565225',
    },
    9144: {
      carro: '9144',
      linha: '153',
      dataHora: '11/07/2024 07:55:22',
      latitude: '-26.86975100',
      longitude: '-49.13152500',
    },
    9149: {
      carro: '9149',
      linha: '15',
      dataHora: '11/07/2024 07:55:35',
      latitude: '-26.92075600',
      longitude: '-49.06758200',
    },
    9150: {
      carro: '9150',
      linha: '11',
      dataHora: '11/07/2024 07:55:40',
      latitude: '-26.92795400',
      longitude: '-49.05728400',
    },
    9152: {
      carro: '9152',
      linha: '31',
      dataHora: '11/07/2024 07:55:36',
      latitude: '-26.93393300',
      longitude: '-49.13057200',
    },
    9153: {
      carro: '9153',
      linha: '12',
      dataHora: '11/07/2024 07:55:39',
      latitude: '-26.92861200',
      longitude: '-49.05755700',
    },
    9159: {
      carro: '9159',
      linha: '124',
      dataHora: '11/07/2024 07:55:40',
      latitude: '-26.86862400',
      longitude: '-49.09819000',
    },
    9161: {
      carro: '9161',
      linha: '32',
      dataHora: '11/07/2024 07:55:40',
      latitude: '-26.90620900',
      longitude: '-49.07558500',
    },
    9164: {
      carro: '9164',
      linha: '507',
      dataHora: '11/07/2024 07:55:34',
      latitude: '-26.92835900',
      longitude: '-49.05745200',
    },
    9171: {
      carro: '9171',
      linha: '32',
      dataHora: '11/07/2024 07:55:41',
      latitude: '-26.91193800',
      longitude: '-49.11282800',
    },
    9172: {
      carro: '9172',
      linha: '123',
      dataHora: '11/07/2024 07:55:39',
      latitude: '-26.86069300',
      longitude: '-49.08606900',
    },
    9173: {
      carro: '9173',
      linha: '31',
      dataHora: '11/07/2024 07:55:34',
      latitude: '-26.91538400',
      longitude: '-49.07295600',
    },
    9188: {
      carro: '9188',
      linha: '12',
      dataHora: '11/07/2024 07:55:31',
      latitude: '-26.89968000',
      longitude: '-49.09798000',
    },
    9190: {
      carro: '9190',
      linha: '32',
      dataHora: '11/07/2024 07:55:35',
      latitude: '-26.93433900',
      longitude: '-49.13034100',
    },
    9193: {
      carro: '9193',
      linha: '12',
      dataHora: '11/07/2024 07:55:31',
      latitude: '-26.88176846',
      longitude: '-49.10324679',
    },
    9194: {
      carro: '9194',
      linha: '30',
      dataHora: '11/07/2024 07:55:49',
      latitude: '-26.92724500',
      longitude: '-49.05729000',
    },
    9196: {
      carro: '9196',
      linha: '12',
      dataHora: '11/07/2024 07:55:38',
      latitude: '-26.92872400',
      longitude: '-49.05741400',
    },
    9303: {
      carro: '9303',
      linha: '403',
      dataHora: '11/07/2024 07:55:27',
      latitude: '-26.96288400',
      longitude: '-49.06484600',
    },
    9304: {
      carro: '9304',
      linha: '603',
      dataHora: '11/07/2024 07:55:34',
      latitude: '-26.85749400',
      longitude: '-49.04816100',
    },
    9310: {
      carro: '9310',
      linha: '302',
      dataHora: '11/07/2024 07:55:35',
      latitude: '-26.93378800',
      longitude: '-49.13109400',
    },
    9312: {
      carro: '9312',
      linha: '32',
      dataHora: '11/07/2024 07:55:38',
      latitude: '-26.91246700',
      longitude: '-49.08026200',
    },
    9314: {
      carro: '9314',
      linha: '30',
      dataHora: '11/07/2024 07:55:36',
      latitude: '-26.92828000',
      longitude: '-49.05741600',
    },
    9446: {
      carro: '9446',
      linha: '81',
      dataHora: '11/07/2024 07:27:27',
      latitude: '-26.79860045',
      longitude: '-49.08510827',
    },
    9449: {
      carro: '9449',
      linha: '601',
      dataHora: '11/07/2024 07:55:39',
      latitude: '-26.86560800',
      longitude: '-49.05745700',
    },
    9453: {
      carro: '9453',
      linha: '81',
      dataHora: '11/07/2024 07:55:44',
      latitude: '-26.79835900',
      longitude: '-49.08456000',
    },
    9461: {
      carro: '9461',
      linha: '804',
      dataHora: '11/07/2024 07:55:27',
      latitude: '-26.73143000',
      longitude: '-49.06911400',
    },
    9464: {
      carro: '9464',
      linha: '616',
      dataHora: '11/07/2024 07:55:37',
      latitude: '-26.91092800',
      longitude: '-49.08158300',
    },
    9468: {
      carro: '9468',
      linha: '111',
      dataHora: '11/07/2024 07:55:40',
      latitude: '-26.85980600',
      longitude: '-49.08338100',
    },
    9470: {
      carro: '9470',
      linha: '154',
      dataHora: '11/07/2024 07:55:44',
      latitude: '-26.87432600',
      longitude: '-49.18350200',
    },
    9476: {
      carro: '9476',
      linha: '80',
      dataHora: '11/07/2024 07:55:32',
      latitude: '-26.81747300',
      longitude: '-49.08948100',
    },
    9478: {
      carro: '9478',
      linha: '904',
      dataHora: '11/07/2024 07:55:41',
      latitude: '-26.91263000',
      longitude: '-49.08060900',
    },
    9480: {
      carro: '9480',
      linha: '80',
      dataHora: '11/07/2024 07:55:42',
      latitude: '-26.86305100',
      longitude: '-49.08223500',
    },
    9482: {
      carro: '9482',
      linha: '122',
      dataHora: '11/07/2024 07:54:44',
      latitude: '-26.80070000',
      longitude: '-49.10928300',
    },
    9486: {
      carro: '9486',
      linha: '31',
      dataHora: '11/07/2024 07:55:08',
      latitude: '-26.92861064',
      longitude: '-49.05736417',
    },
    9487: {
      carro: '9487',
      linha: '508',
      dataHora: '11/07/2024 07:55:40',
      latitude: '-26.92342900',
      longitude: '-49.06234100',
    },
    9490: {
      carro: '9490',
      linha: '402',
      dataHora: '11/07/2024 07:50:54',
      latitude: '-26.97274715',
      longitude: '-49.09372059',
    },
    9491: {
      carro: '9491',
      linha: '507',
      dataHora: '11/07/2024 07:54:55',
      latitude: '-26.90195326',
      longitude: '-49.03628997',
    },
    9492: {
      carro: '9492',
      linha: '405',
      dataHora: '11/07/2024 07:33:35',
      latitude: '-26.99532095',
      longitude: '-49.07100898',
    },
    9497: {
      carro: '9497',
      linha: '605',
      dataHora: '11/07/2024 07:55:35',
      latitude: '-26.87993000',
      longitude: '-49.06748800',
    },
    9500: {
      carro: '9500',
      linha: '308',
      dataHora: '11/07/2024 07:55:37',
      latitude: '-26.90269700',
      longitude: '-49.15075000',
    },
    9502: {
      carro: '9502',
      linha: '704',
      dataHora: '11/07/2024 07:55:31',
      latitude: '-26.91371200',
      longitude: '-49.14029900',
    },
    9713: {
      carro: '9713',
      linha: '305',
      dataHora: '11/07/2024 07:55:38',
      latitude: '-26.94287800',
      longitude: '-49.12847100',
    },
    9717: {
      carro: '9717',
      linha: '509',
      dataHora: '11/07/2024 07:55:34',
      latitude: '-26.93109600',
      longitude: '-49.05948200',
    },
    9720: {
      carro: '9720',
      linha: '802',
      dataHora: '11/07/2024 07:55:43',
      latitude: '-26.78440000',
      longitude: '-49.08334200',
    },
  },
  {
    9101: {
      carro: '9101',
      linha: '10',
      dataHora: '11/07/2024 08:02:42',
      latitude: '-26.87024100',
      longitude: '-49.09507900',
    },
    9102: {
      carro: '9102',
      linha: '10',
      dataHora: '11/07/2024 08:03:13',
      latitude: '-26.94416200',
      longitude: '-49.07192800',
    },
    9103: {
      carro: '9103',
      linha: '10',
      dataHora: '11/07/2024 08:03:15',
      latitude: '-26.90565800',
      longitude: '-49.07521500',
    },
    9116: {
      carro: '9116',
      linha: '10',
      dataHora: '11/07/2024 08:03:04',
      latitude: '-26.96217500',
      longitude: '-49.06539500',
    },
    9118: {
      carro: '9118',
      linha: '10',
      dataHora: '11/07/2024 08:03:04',
      latitude: '-26.87148900',
      longitude: '-49.09119400',
    },
    9120: {
      carro: '9120',
      linha: '10',
      dataHora: '11/07/2024 08:03:03',
      latitude: '-26.92284500',
      longitude: '-49.06378400',
    },
    9123: {
      carro: '9123',
      linha: '405',
      dataHora: '11/07/2024 08:03:02',
      latitude: '-26.97983200',
      longitude: '-49.07510100',
    },
    9129: {
      carro: '9129',
      linha: '15',
      dataHora: '11/07/2024 08:03:01',
      latitude: '-26.90123300',
      longitude: '-49.07672600',
    },
    9132: {
      carro: '9132',
      linha: '31',
      dataHora: '11/07/2024 08:03:01',
      latitude: '-26.91589600',
      longitude: '-49.10872300',
    },
    9134: {
      carro: '9134',
      linha: '300',
      dataHora: '11/07/2024 08:03:11',
      latitude: '-26.88000800',
      longitude: '-49.10134900',
    },
    9135: {
      carro: '9135',
      linha: '123',
      dataHora: '11/07/2024 08:02:56',
      latitude: '-26.87069000',
      longitude: '-49.09507100',
    },
    9140: {
      carro: '9140',
      linha: '11',
      dataHora: '11/07/2024 08:01:56',
      latitude: '-26.89709561',
      longitude: '-49.07160376',
    },
    9144: {
      carro: '9144',
      linha: '153',
      dataHora: '11/07/2024 08:03:01',
      latitude: '-26.86965500',
      longitude: '-49.10561700',
    },
    9150: {
      carro: '9150',
      linha: '11',
      dataHora: '11/07/2024 08:02:57',
      latitude: '-26.92440900',
      longitude: '-49.05612200',
    },
    9152: {
      carro: '9152',
      linha: '30',
      dataHora: '11/07/2024 08:03:15',
      latitude: '-26.93015500',
      longitude: '-49.12004200',
    },
    9153: {
      carro: '9153',
      linha: '12',
      dataHora: '11/07/2024 08:02:59',
      latitude: '-26.92312900',
      longitude: '-49.05597600',
    },
    9161: {
      carro: '9161',
      linha: '32',
      dataHora: '11/07/2024 08:03:00',
      latitude: '-26.91383700',
      longitude: '-49.08332900',
    },
    9164: {
      carro: '9164',
      linha: '507',
      dataHora: '11/07/2024 08:02:53',
      latitude: '-26.90807000',
      longitude: '-49.04165800',
    },
    9171: {
      carro: '9171',
      linha: '32',
      dataHora: '11/07/2024 08:03:01',
      latitude: '-26.91143800',
      longitude: '-49.08943700',
    },
    9173: {
      carro: '9173',
      linha: '31',
      dataHora: '11/07/2024 08:03:15',
      latitude: '-26.91578600',
      longitude: '-49.08816600',
    },
    9188: {
      carro: '9188',
      linha: '12',
      dataHora: '11/07/2024 08:03:10',
      latitude: '-26.87190000',
      longitude: '-49.09568500',
    },
    9190: {
      carro: '9190',
      linha: '32',
      dataHora: '11/07/2024 08:03:14',
      latitude: '-26.93163500',
      longitude: '-49.13504300',
    },
    9193: {
      carro: '9193',
      linha: '12',
      dataHora: '11/07/2024 08:03:08',
      latitude: '-26.90336600',
      longitude: '-49.09801000',
    },
    9194: {
      carro: '9194',
      linha: '30',
      dataHora: '11/07/2024 08:03:08',
      latitude: '-26.92866600',
      longitude: '-49.05751400',
    },
    9303: {
      carro: '9303',
      linha: '403',
      dataHora: '11/07/2024 08:03:06',
      latitude: '-26.98141800',
      longitude: '-49.07564100',
    },
    9304: {
      carro: '9304',
      linha: '603',
      dataHora: '11/07/2024 08:03:12',
      latitude: '-26.86276200',
      longitude: '-49.05334400',
    },
    9310: {
      carro: '9310',
      linha: '302',
      dataHora: '11/07/2024 08:03:14',
      latitude: '-26.93211000',
      longitude: '-49.13878600',
    },
    9312: {
      carro: '9312',
      linha: '32',
      dataHora: '11/07/2024 08:02:59',
      latitude: '-26.91215900',
      longitude: '-49.09845500',
    },
    9314: {
      carro: '9314',
      linha: '30',
      dataHora: '11/07/2024 08:02:55',
      latitude: '-26.92466500',
      longitude: '-49.05618600',
    },
    9446: {
      carro: '9446',
      linha: '81',
      dataHora: '11/07/2024 08:03:13',
      latitude: '-26.84799000',
      longitude: '-49.08372400',
    },
    9449: {
      carro: '9449',
      linha: '606',
      dataHora: '11/07/2024 08:02:55',
      latitude: '-26.87998200',
      longitude: '-49.06759900',
    },
    9453: {
      carro: '9453',
      linha: '81',
      dataHora: '11/07/2024 08:03:04',
      latitude: '-26.79835900',
      longitude: '-49.08456000',
    },
    9458: {
      carro: '9458',
      linha: '701',
      dataHora: '11/07/2024 08:03:10',
      latitude: '-26.91168600',
      longitude: '-49.08089600',
    },
    9461: {
      carro: '9461',
      linha: '804',
      dataHora: '11/07/2024 08:03:06',
      latitude: '-26.77408300',
      longitude: '-49.08239500',
    },
    9468: {
      carro: '9468',
      linha: '111',
      dataHora: '11/07/2024 08:03:00',
      latitude: '-26.85035200',
      longitude: '-49.07414800',
    },
    9470: {
      carro: '9470',
      linha: '154',
      dataHora: '11/07/2024 08:03:03',
      latitude: '-26.88278400',
      longitude: '-49.15361000',
    },
    9476: {
      carro: '9476',
      linha: '80',
      dataHora: '11/07/2024 08:03:10',
      latitude: '-26.79802900',
      longitude: '-49.08459700',
    },
    9480: {
      carro: '9480',
      linha: '80',
      dataHora: '11/07/2024 08:03:03',
      latitude: '-26.83131500',
      longitude: '-49.09053600',
    },
    9482: {
      carro: '9482',
      linha: '122',
      dataHora: '11/07/2024 08:03:07',
      latitude: '-26.83377033',
      longitude: '-49.10623561',
    },
    9486: {
      carro: '9486',
      linha: '31',
      dataHora: '11/07/2024 08:03:03',
      latitude: '-26.92432146',
      longitude: '-49.05607562',
    },
    9490: {
      carro: '9490',
      linha: '402',
      dataHora: '11/07/2024 08:00:04',
      latitude: '-26.97160847',
      longitude: '-49.09655962',
    },
    9491: {
      carro: '9491',
      linha: '507',
      dataHora: '11/07/2024 08:02:59',
      latitude: '-26.91607800',
      longitude: '-49.05777300',
    },
    9492: {
      carro: '9492',
      linha: '405',
      dataHora: '11/07/2024 08:03:09',
      latitude: '-26.96462000',
      longitude: '-49.07295400',
    },
    9497: {
      carro: '9497',
      linha: '605',
      dataHora: '11/07/2024 08:03:15',
      latitude: '-26.88242400',
      longitude: '-49.07015300',
    },
    9500: {
      carro: '9500',
      linha: '308',
      dataHora: '11/07/2024 08:02:58',
      latitude: '-26.90085000',
      longitude: '-49.14540100',
    },
    9502: {
      carro: '9502',
      linha: '704',
      dataHora: '11/07/2024 08:03:09',
      latitude: '-26.89689800',
      longitude: '-49.15197200',
    },
    9709: {
      carro: '9709',
      linha: '422',
      dataHora: '11/07/2024 08:03:08',
      latitude: '-26.96202700',
      longitude: '-49.06560800',
    },
    9713: {
      carro: '9713',
      linha: '311',
      dataHora: '11/07/2024 08:02:58',
      latitude: '-26.94249800',
      longitude: '-49.12835100',
    },
    9720: {
      carro: '9720',
      linha: '802',
      dataHora: '11/07/2024 08:02:43',
      latitude: '-26.76237600',
      longitude: '-49.07458900',
    },
  },
];
1 Like
2

How do I reproduce the issue with the script? I tried running it with Node v18.20.4 and QDB 8.0.1, but all column values seem to be populated fine.

Also, this while loop is redundant:

let stillHasDataInBuffer = false;
  do {
    stillHasDataInBuffer = await sender.flush();
    console.log('stillHasDataInBuffer: ', stillHasDataInBuffer);
  } while (stillHasDataInBuffer);

It’s enough to call await sender.flush() only once.

Other than that, you may use the same sender instance between saveData calls in the for (const cycleData of allCycleData) loop. This is more efficient since this way the same HTTP connection will be reused.

3

Since you couldn’t reproduce the issue, I tried a different configuration in my docker-compose. Initially, I was running the QuestDB image with a volume mounted to a Windows folder. This setup caused the issue. However, when I removed the volume, it worked as expected.

Using just the await sender.flush(); without the volume worked fine too.

Could mounting the QuestDB folder to a Windows directory be causing this problem?

This was the docker-compose that I see the issue.

version: '3.7'

services:
  questdb:
    image: questdb/questdb
    container_name: questdb
    restart: always
    ports:
      - 9000:9000
      - 9009:9009
      - 8812:8812
      - 9003:9003
    volumes:
      - ./.questdb:/root/.questdb
    environment:
      - QDB_LOG_W_STDOUT_LEVEL=ERROR
      - QDB_LOG_W_FILE_LEVEL=ERROR
      - QDB_LOG_W_HTTP_MIN_LEVEL=ERROR
      - QDB_SHARED_WORKER_COUNT=2
4

Yes, it could be the case that the overlay file system used by Docker doesn’t support all system calls that we need in QuestDB. Here is the list of supported file systems.