Related Model Collections - Mappable Data and ElasticSearch Datatypes

[drmmr763]

Hello

Thank you for the hardwork on this package it is exactly what I was hoping to find to get an elastic based index up for my app.

I have a document with attached models, that could be a HasOne or HasMany relationship. I want to include all the attached entities as part of the document to be indexed. I was hoping I could just define for each model the mappableAs function and as the indexing was taking place, it would jump through each of the relationships, get the mapped data and field types as defined and map those fields accordingly.

That doesn't seem to happening. When I set up the nested field type, it does include the model I want, but its not mapping the field types. I did customize the toSearchableArray method on a child model, and even though I only returned a single field, the request to index is getting all the related model's fields. So, it doesn't seem to be referencing mappableAs or toSearchableArray when navigating through the root entity attach the nested models.

Example code of how I have things configured:

Candidate.php

class Candidate extends Model implements HasAttachedSkills, Explored
{
   /**
     * @return HasMany
     */
    public function candidateSkills(): HasMany
    {
        return $this->hasMany(
            CandidateHasSkill::class,
            'candidate_id',
            'id'
        )->whereNull('candidate_has_skills.deleted_at')
            ->leftJoin('skills', 'skills.id', '=', 'candidate_has_skills.skill_id')
            ->whereNull('skills.deleted_at');
    }

    public function mappableAs(): array
    {
        return [
            'id' => 'keyword',
               // how do I define the nested object's field type mappings? it doesn't seem to be reading off CandidateHasSkill->mappableAs()
            'candidate_skills' => 'nested' // this would be a collection of CandidateHasSkill
           
        ];
    }
 
    /**
     * @return string
     */
    public function searchableAs(): string
    {
        return 'candidates_index'; // this is the only real document I want to index, the other models are attached to this
    }
}

CandidateHasSkill.php

class CandidateHasSkill extends Model implements Explored
{
    /**
     * @return BelongsTo
     */
    public function skill() : BelongsTo
    {
        return $this->belongsTo(Skill::class, 'skill_id', 'id');
    }

  /**
     * Laravel model fields to ElasticSearch data type mappings.
     * @return string[]
     */
    public function mappableAs(): array
    {
        return [
            'id' => 'keyword',
            'skill_id' => 'keyword',
            'candidate_id' => 'keyword',
            'skill' => 'nested'
        ];
    }

     /**
     * Get the indexable data array for the model.
     *
     * @return array<string, mixed>
     */
    public function toSearchableArray(): array
    {
        $array = $this->toArray();

        return [
            'id' => $array['id'] // should only be returning the id for this model, but entire model and attached fields are sent to Elastic. 
        ];
    }
}

Skill.php

class Skill extends Model implements Explored
{
    use HasTimestamps;
    use SoftDeletes;
    use TraitAuthorship;
    use Searchable;

   /**
     * Laravel model fields to ElasticSearch data type mappings.
     * @return string[]
     */
    public function mappableAs(): array
    {
        return [
            'id' => 'keyword',
            'name' => 'text',
            'label' => 'text'
        ];
    }

      /**
     * Get the indexable data array for the model.
     *
     * @return array<string, mixed>
     */
    public function toSearchableArray(): array
    {
        $array = $this->toArray();

        return [
            'id' => $array['id'] // should only be returning the id for this model, but entire model and attached fields are sent to Elastic. 
        ];
    }
}

I guess my specific questions / issues are:

1. Is there a way to customize the attached model properties?
2. Can that way be within the model its self, or do I have to do it all from the root model?
3. I assume if I were to customize using either prepare or toSearchableArray on my root document array, I might be able to achieve what I'm wanting, but I was hoping to have the concerns split across the model classes.
4. If the only way to do it is on the root document, how would that look for say a collection vs a single field?
5. How do I define the nested object's property field type mappings? I have defined it on the model but on the root object I just do the nested type.

[drmmr763]

Small followup, I did confirm I could control the mapping from the root model:

/**
     * Get the indexable data array for the model.
     *
     * @return array<string, mixed>
     */
    public function toSearchableArray(): array
    {
        $array = $this->toArray();

        $array['candidate_skills'] = $this->candidateSkills->map(function ($item) {
            return $item->toSearchableArray();
        });

        return $array;
    }

Its not quite as elegant as I'd like but I recognize that might be a scout limitation. Still if there is a way to do it more per-model rather than the root model I'd love to know it.

[Jeroen-G]

Thank you for the elaborate description, really! 👏

I dove into the source and this is the culprit: https://github.com/Jeroen-G/Explorer/blob/master/src%2FApplication%2FOperations%2FBulk%2FBulkUpdateOperation.php#L62

It does nothing to see what happens in the root, it expects a plain array in return. Apparently you are the first to encounter this or be bothered by it😁

Unless you know of a smart solution I think you will have to do everything as you showed in the root toSearchableArray.

[drmmr763]

Thanks @Jeroen-G for confirming I wasn't missing anything!

I guess that explains why it doesn't work. For now I've gone ahead and set up my root document to call the toSearchableArray on child relations as needed.

I did want to follow up on one thing though: the nested object field mappings are also not being used to create the index in Elastic.

For example, in the code I give above with something like a Candidate, CandidateHasSkill, and Skill object, In the Candidate model, I can use mappableAs to define the field types I want in Elastic. I then also define candidate_has_skill as a nested field.

How do tell Elastic what the field types for candidate_has_skill are to be? it is doing automatically right now and getting it a bit wrong. For example its using integer or long for IDs when per Elastic's docs it really should be keyword.

https://www.elastic.co/guide/en/elasticsearch/reference/current/keyword.html

Is there a way to define a nested object's field mappings?

[Jeroen-G]

There is a bit on nested mappings here: https://jeroen-g.github.io/Explorer/mapping.html which might help you!

I think this could work:

return [
    'id' => 'keyword',
    'candidate_skills' => [
        'name' => 'keyword'
    ]
];