unique option is not working for text field ?

[geminigeek]

Bug Report

i just installed a fresh copy of payload cms just added a new field 'userName' like below to Users.js

{ name: "userName", type: "text", index: true, required: true, unique: true, }, the new user creation should not go through, if we have same userName , but the unique checks are not happening ? i am able to add new users with same user name ?

however the required option is working.

also no index is created even while i used index key above ?

new record with same userName should not go through, also an index should be created with unique flag .

i am using Win 10 , Mongodb 5.0.1 , node 16.13.1

[DanRibbens]

It most definitely should create that unique index for you.

It should try to make the index everytime Payload starts if it isn't there already.

If you drop the collection in your database and start again is it created for you?

[geminigeek]

hi Dan,

thanks for quick reply.

i just did a a test of same on tags collection. dropped tags and restarted below is my Tags.js.

still i can add same tags and no unique index created ?

const Tags = {
  slug: "tags",
  admin: {
    useAsTitle: "name",
  },
  access: {
    read: () => true,
  },
  fields: [
    {
      name: "name",
      type: "text",
      unique: true,
      index: true,
    },
  ],
  timestamps: false,
}

export default Tags

[DanRibbens]

I don't see anything wrong here. You could remove index: true as it is redundant when also using unique. It should not make a difference, but could be good to rule out that isn't an issue.

Have you customized your mongo connection settings? Can you share those settings if so?

Otherwise it could be a version compatibility issue with using MongoDB 5. I haven't tested Payload it out there yet and will have to dig in a little deeper there, though it should work fine. We're using Mongoose v6 which is supposed to be compatible with MongoDB 5.

[geminigeek]

Hi,

i removed the index:true , dropped collection again no change. i haven't customized any mongo settings here is from .env

MONGODB_URI=mongodb://localhost/payloadcms-test
PAYLOAD_SECRET=secret

i have used Mongoose v6 with mongodb 5.0 , but i am sure mongoose doesn't handle indexes for unique, if payloadcms is not invoking indexes as settings change in collection files , mongoose will not make them !!

you guys relying on mongoose to make indexes will not work IMHO out of the box, when settings change.

[geminigeek]

reference : https://mongoosejs.com/docs/faq.html#unique-doesnt-work

MongoDB persists indexes, so you only need to rebuild indexes if you're starting with a fresh database or you ran db.dropDatabase(). In a production environment, you should [create your indexes using the MongoDB shell](https://docs.mongodb.com/manual/reference/method/db.collection.createIndex/) rather than relying on mongoose to do it for you. The unique option for schemas is convenient for development and documentation, but mongoose is not an index management solution

[DanRibbens]

We've looked in to this a few times in the past when indexes didn't quite match up. We've kind of skirted this one and gotten away with doing it this way for a while, but it isn't perfect. We need to set aside some time to do it the right way as you've pointed out.

In the short term, the workaround is to manually create indexes that aren't being created as intended. This is obviously not ideal but in most cases we are seeing indexes are made where needed.

I propose we add a new command, perhaps calling it payload db:index. This will be similar to what we have done with payload generate:types that loops the collections and fields in purpose built script. The script would loop all the fields and create indexes outside of the mongoose schemas. On any large data sets, adding an index at startup will be problematic so we'll add a flag for making it optional at startup. To be debated is whether to create indexes on start up or not, maybe this depends on node_env or another option.

I'd also like to see an option that controls checking vs actually creating the indexes. Possibly naming the flag --dry-run or --check. This would allow you to safely report out missing indexes separately from creating them on large datasets where that could be a problem.

How does this sound? Any questions or things to add?

[geminigeek]

Hi Dan,

Great suggestion of --dry-run , i think in dev ENV/debug mode it should list the indexes that are miss match or need to be created and a message/command, suggesting a flag to be added for auto index creation , it must also look up total records count for the respective index like up till 1000 and also suggest the performance issue while enabling the index flag.

i am still new to Node/JS so its up for suggestion and discussion .

Lastly i must say, i am testing Payload for just 1 day , i am fairly impressed with the possibility it offers, great work!

[DanRibbens]

I haven't been able to prioritize writing the script to add indexes outside of mongoose yet. If anyone has time I would be able to give it some attention for reviewing PRs and testing.

One thing to add to the conversation is that there is a mongoose setting for autoIndex and defaults to true. This would be helpful when you really don't want index creation to occur on server start and would look something like this in a Payload project:

payload.init({
  mongoURL: process.env.MONGO_URL,
  mongoOptions: {
    autoIndex: !(process.env.NODE_ENV === 'production'),
  },
  // ...
})

[HopeFaith2012]

payload.init({ mongoURL: process.env.MONGO_URL, mongoOptions: { autoIndex: !(process.env.NODE_ENV === 'production'), }, // ... })

[ayusshrathore]

Hey @DanRibbens, I've tried with the mongoOptions that you mentioned, but still unique is still not working for the text fields. Could you please state a workaround for this?

[ayusshrathore]

Hey @geminigeek, were you able to store unique text in the collection? If so, could you please enlighten me with the same.

[geminigeek]

Hey @geminigeek, were you able to store unique text in the collection? If so, could you please enlighten me with the same.

i am making indexes manually for now from mongo shell

[ayusshrathore]

Hey @geminigeek, were you able to store unique text in the collection? If so, could you please enlighten me with the same.

i am making indexes manually for now from mongo shell

Okay, thanks✌🏻

[DanRibbens]

I was able to recreate the problem where indexes are not being made on start and found it to be a difference between mongo 4.4.15 and 5.0.3.

This is the repo I used to run different versions of mongo using docker.

The interesting thing that I found is that our point field which calls up index creation after creating the schema objects in mongoose actually does work. The line in which we call index creation for some fields is here: https://github.com/payloadcms/payload/blob/b722bed24f752742e04d002cb37a4a649b98ab16/src/mongoose/buildSchema.ts#L118

We are looking into the issue more to determine if calling schema.index() will resolve the problems for all index creation.

[DanRibbens]

With #791 we now create indexes on startup for mongoDB 5.

We are working on adding the function that can be used in production that will give more visibility and intention.

[andwrobs]

^ thanks @DanRibbens . Q- is there a way to tap into this functionality today?

Or by

"We are working on adding the function that can be used in production that will give more visibility and intention."

... do you mean a function to make it accessible at all in the config?

Separately, A maybe interesting case I want to flag is that I'm using something similar to your public demo,

    {
      name: 'internalTitle',
      unique: true,
      type: 'text',
      label: 'Title',
      defaultValue: `some default`,
      hooks: {
        beforeChange: [
           // custom hook function to make title from properties
           populateInternalTitle
        ]
    }

... and it doesn't seem to enforce any uniqueness on the value generated by the beforeChange hook.

We publish daily content, so I wanted to automatically generate the sortable human-legible CMS title based on publishDate date field & the collection name. Unique to not allow more than one entry per day.

Have updated to 1.0.9.

[henrytrianta]

Hey @geminigeek, were you able to store unique text in the collection? If so, could you please enlighten me with the same.

we tried and it works with adding option localized: false in the text field. we use mongodb atlas as a source of database

[DanRibbens]

A related bug #859 was fixed in release v1.0.12 that had to do with unique indexes. All the problems with indexing should be resolved now. @andwrobs, this covers the bug you pointed out in the demo.

We also decided to not do anything in the short term regarding the idea to add a payload script for building indexes. The reason for this decision is that:

1. The functionality is already a part of mongoose with diffIndexes() and ensureIndexes() which can be called on the Model. See https://mongoosejs.com/docs/api.html#model_Model-diffIndexes and https://mongoosejs.com/docs/api.html#model_Model-ensureIndexes
2. It could be built into Payload, but it belongs under a subset of features that would essentially be a migration tool that is more robust.

I'm going to close this issue for now because I think we have addressed everything in the thread. Please open any new issues or discussions for new features as needed. Thanks everyone who contributed!

[johannesschaffer]

It still doesn't work! I can still save multiple entries which have all the same text field.

Edit: Solved it

• Drop the collection in MongoDB
• (? Delete the index)
• Restart Payload

[saaymeen]

I noticed that when setting the unique field to true on a collection that is already used in production (or already added to Mongo), the change has no effect. I can still save multiple entries to the collection, all having a text field with the same value. localized or index do not affect this behavior.

Dropping the collection in production is not an option (of course?).

[email protected]

[jmikrut]

Well, we are declaring Mongoose's autoIndex: true, so I do believe that restarting your server should automatically create indices for newly updated fields. The problem may be that if you create a new unique index, and the index is not able to be created because of existing data interfering, then it might fail.

@saaymeen is this your issue do you think?

[iakshay]

Ran into same issue, thinking if it might be possible to have an error log for when index creation fails.

P.S - Discovered this issue, from the link in tests. Thank you!