Backup validation fails when using mongodb "View"s

[erlestor]

Im trying to run a backup with validation turned on. However I get the following two errors:

The database I'm backing up contains mostly normal collections. Except one time series collection. Which is stored as a "datapoints" view and "system.buckets.datapoints" collection. Validation only fails for "datapoints" and "system.views". Which are both views obviously.

I can backup without validation. Then do a restore with validation. And everything works fine. So there's something specific to validating views from a backup that is going wrong

I can't just turn off validation as then restore stops working. See #134

I would write a PR, but I can't write go and want confirmation that I'm not doing anything else wrong first

[maxisam]

please test maxisam/mgob:dev.339 thx!

https://hub.docker.com/repository/docker/maxisam/mgob/tags/dev.339/sha256:56b23a9a83fdfac59c4c95e7c8e9f636c633dc7353fac4d60a4faa3f1a15f69d

[maxisam]

@erlestor any chance to test it?

[erlestor]

I didn't see that you provided a possible fix already. Thanks! I'll test it out soon

[erlestor]

I turned on validation and tested with both on demand and scheduled backup. They both give the same two errors:

"system.profile" is the collection mongo uses when profiling is turned on. We use it for monitoring in grafana. And "datapoints" is our timeseries collection. So it is a mongodb view.

I'll just dump my config so you can see if there's something I'm doing wrong:

# ./compose.test.yml
   mongo-backup:
      container_name: test-mongo-backup
      # image: maxisam/mgob:2.0.28
      image: maxisam/mgob:dev.339
      volumes:
         - mongo-backup:/storage
         - ./mongo-backup.yml:/config/mongo.yml
      command:
         - "-LogLevel=info"
      environment:
         MONGO__TARGET_PORT: ${MONGO_PORT}
         MONGO__TARGET_USERNAME: ${MONGO_USERNAME}
         MONGO__TARGET_PASSWORD: ${MONGO_PASSWORD}
         MONGO__VALIDATION_DATABASE_PORT: ${MONGO_PORT}
         MONGO__VALIDATION_DATABASE_USERNAME: ${MONGO_USERNAME}
         MONGO__VALIDATION_DATABASE_PASSWORD: ${MONGO_PASSWORD}
      depends_on:
         - mongo
      networks:
         - prod
      # TODO: remove. just for testing
      ports:
         - 8090:8090

# ./mongo-backup.yml
scheduler:
   # cron: "0 21 */1 * *" # every day at 21:00 UTC -> 23:00 CET
   cron: "21 13 */1 * *" # every day at 21:00 UTC -> 23:00 CET
   retention: 3 # amount of backups
   timeout: 60 # minutes

target:
   host: "mongo"
   port: 0
   username: ""
   password: ""
   database: "prod"
   params: "--authenticationDatabase admin "
   noGzip: false

retry:
   attempts: 1
   backoffFactor: 1
   # attempts: 3
   # backoffFactor: 60
# -----------------------------------------------------------------------------
# TODO: turn on validation when mongodb views are supported in mgob
# - see: https://github.com/maxisam/mgob/issues/167
validation:
   database:
      host: "mongo"
      port: 0
      username: ""
      password: ""
      database: "test_restore" # should never match the prod database name
      params: "--authenticationDatabase admin --drop"
      noGzip: false

[maxisam]

I am very busy in Nov because of the black friday. But I will try to check it when i have time. Also maybe you can share how you use it and I can create the same setup. Do you have any reference link I can read about your setup?

[erlestor]

Sure! What do you need exactly? Would the binary file from mongodump suffice? Along with the config files above and which commands i use to test it?

[maxisam]

more like how do you setup that? any ref you can find for me to try?