stagehand icon indicating copy to clipboard operation
stagehand copied to clipboard

Remove `initFromPage` to make local more configurable

Open kamath opened this issue 11 months ago • 8 comments

We should refactor our init scripts to make what's currently possible in initFromPage to run directly from init without the need for a separate method. This means being able to reuse contexts and other necessary configuration in regular stagehand initialization

kamath avatar Dec 03 '24 02:12 kamath

nice!

RicoP avatar Dec 05 '24 15:12 RicoP

Added example to show blending using robot.glb model.

https://github.com/user-attachments/assets/ad1feb31-c54d-47b9-8dcf-1029b1620690

Kirandeep-Singh-Khehra avatar Dec 07 '24 17:12 Kirandeep-Singh-Khehra

@Kirandeep-Singh-Khehra This is a great addition! I find the function name UpdateModelAnimationBonesWithBlending() quite long, maybe it can be shortened a bit? I'm thinking about UpdateModelAnimationPro(), that, despite not being so clear, it follows raylib naming conventions for advance versions of some functions (UpdateCameraPro(), DrawTexturePro(), DrawTextPro()...).

In the same line, UpdateModelAnimationBones() is the GPU alternative to UpdateModelAnimation(), maybe it can be simplified to just one function and enable GPU-skinning with a compile flag? Not sure how useful is having both methods available.

That way, UpdateModelAnimationPro() could implement blending internally in CPU/GPU using the same compile flag.

raysan5 avatar Dec 08 '24 12:12 raysan5

While I agree that we need interpolation/blending support, I feel that this simple API does not go far enough.

  1. I would like to see the ability to update the bones separate from the model. I may have many instances of a model at different animation states, I want to keep my own set of 'current' bones with my instance structure and not with the model. So to me an optimal function would take the output bone matrices as an argument.
  2. I would like to be able to pick what I apply the interpolated frame to, the vertices or upload them to the GPU.

I don't mind having a basic function that just updates the model's internal bone state, but I would like the lower level functions to be accessible for people who need more than a simple single model use case.

JeffM2501 avatar Dec 08 '24 22:12 JeffM2501

@raysan5 @JeffM2501 Thanks guys for sharing your feedback and concerns,

When i started using raylib i was hoping for something like simple utility to work with skeleton and forward kinematics. And both these things were way complex. And i was unable to make a good looking 3d game without adding something to animations.

But as Ray asked to unify animation functions. I discarded/upgraded this code and tried to create a one stop solution and i implemented most of it. Just minor adjustments are needed.

I completely agree that the idea discussed below is lot complex(but code will be readable and simple to debug, i assure). But it will add things like blending, interpolation, split body animation. single call function for more complex animations(like add all animations in single call and user only need calculate weights(discussed below)) But if we consider it to be a function at low level. Then it can do a lot on animation side.(see the last code block in this comment)

The idea is to create function UpdateModelAnimationPro(). But for now lets consider only bones UpdateModelAnimationBonesPro(Model, BoneMask, int, int, .../*va_args*/). with paramers including:

  1. Model model: Model to use.
  2. BoneMask mask: Just a fancy name for bitmap. Which stores list of 0 or 1. Useful for model splitting. Like upper body split and lower body split.
  3. int flag: Any config flags. Like ANIM_PRO_INVERT_MASK(to invert the mask before using it) and other flags etc etc. Intended use is like ANIM_PRO_INVERT_MASK | ANIM_PRO_FLAG2 | ANIM_PRO_FLAG3. Not yet planned what could be added.
  4. int animCount: Number of animations range from 1 to N.
  5. ...: va_arg list consisting of multiple ModelAnimation anim, int frame, double weight(va_list promotes float to double and may introduce some issues).

Example i used:

UpdateModelAnimationBonesPro(model, fullBodyMask, 0, 2,
                              /* Anim       |  Frame         |   Weight            */
                              anims[indexX], animFrameCounter, (double)abs(motion.x),
                              anims[indexY], animFrameCounter, (double)abs(motion.y)
                              /* More lines here */
                              );

BoneMask is implemented but not yet used in function.

I tested passing 1,2 and 3 animations and it passed with flying colors. (As far as i can see on screen).

We can also pass ModelAnimation array then int array for frames and double or float array for weights. This will make usage more complex but using va_list will make no-compile time warnings or linting or auto completion.

Then we can replace/redefine UpdateModelAnimationBones() and UpdateModelAnimationBonesWithBlending() with

#define UpdateModelAnimationBones(model, anim, frame) UpdateModelAnimationBonesPro(model, FullBodyMask(), 0, 1, anim, frame, 1)

#define UpdateModelAnimationBonesWithMask(model, animA, frameA, animB, frameB, blendFactor) UpdateModelAnimationBonesPro(model, FullBodyMask(), 0, 2, animA, frameA, (1-blendFactor), animB, frameB, blendFactor)

// This one will be interesting
void UpdateModelAnimationBonesWithTime(model, anim, time) {
  int frameA = time / GLTF_ANIMDELAY;
  int frameB = time / GLTF_ANIMDELAY;

  float blendFactor = (time % GLTF_ANIMDELAY) / GLTF_ANIMDELAY;
  UpdateModelAnimationBonesPro(model, FullBodyMask(), 0, 2, 
                               anim, frameA, (1 - blendFactor),
                               anim, frameB, blendFactor)
}

// ... or many more such animation stuff can be done with this one function.

If its not possible to have this then ... (Just let me know) .Or Do we have plans for official plugin system (separate repo for bunch of simple header files) to add such advanced functionality to raylib.

I am also planning to work on state machine after this for animation but State machine will be lot less useful when added to raylib. So, it must be separate repo. But that's the story of another day.

Looking forward for some feedback.

Kirandeep-Singh-Khehra avatar Dec 09 '24 15:12 Kirandeep-Singh-Khehra

That seems very complicated. I'm not a fan of the bitmask thing, as I don't think people will get it. I am also no a fan of combining interpolation and blending in the same base API.

This makes me wonder if a full animation system should be made external to raylib as a drop in.

I would take blending out of this feature to start with. While I get that it's similar, I think it complicates something that should be a simple API (prevent feature creep).

My thoughts are that we should add the following new API functions to start with.

void UpdateModelAnimationBonesLerp(Model model, ModelAnimation anim, int frame1, int frame2, float param);  // computes a new set of bone matricies between two frames.

void UpdateModelVertsToCurrentBones(Model model); // takes the current set of bone matrices and applies to them to mesh verts (CPU Animation)

These would give us basic interpolation using a simple API that works for both GPU and CPU users without code duplication.

Someone doing GPU animation would just call UpdateModelAnimationBonesLerp and then upload the bones to the shader. Someone doing CPU animation would need to also call UpdateModelVertsToCurrentBones to apply the bones to the verts.

I think it's important for people doing CPU animation to know that they are modifying vertex data.

I don't think there is any benefit to locking in a 'pro' version of any function right now. I fear we may 'burn' the pro name on something that's not useful just to 'get something in'. We do not need to rush this.

Blending is a big subject with a lot of requirements. I think to start for blending we should open a discussion about what those requirement and use cases are before we design any APIs. I think that for blending we are really going to need a way to define a model skeleton outside of the model structure and update it separately, then apply it back to the meshes, and that's a bigger design, something I feel that is outside the scope of a simple single PR. My fear is that a purely internal solution will be lacking in some regards, so an external drop in solution may be best (with API changes to support the access it needs). But we for sure need to design the API properly first, not just throw code around.

JeffM2501 avatar Dec 09 '24 16:12 JeffM2501

Updated the PR with functions suggested by JeffM2501. Example is also updated. To have both CPU and GPU skinning along with comments that user viewer can follow to make it use GPU skinning.

Modified UpdateModelAnimation() (It had the code to update verts) to use UpdateModelVertsToCurrentBones().

Tested GPU and CPU version and other examples using older UpdateModelAnimation function.

void UpdateModelAnimationBonesLerp(Model model, ModelAnimation anim1, int frame1, ModelAnimation anim2, int frame2, float param);  // computes a new set of bone matricies between two frames.

void UpdateModelVertsToCurrentBones(Model model); // takes the current set of bone matrices and applies to them to mesh verts (CPU Animation)

Kirandeep-Singh-Khehra avatar Dec 11 '24 17:12 Kirandeep-Singh-Khehra

Please see the discussion here https://github.com/raysan5/raylib/discussions/4606 We need to start with the lowest level of API, not highest feature level. There are many other use cases that the low level API needs to support other than blending.

I honestly don't think that blending should be built into the core of raylib at all, but be a drop in lib like rayGui.h using the core API.

While I appreciate your enthusiasm for this feature, having this PR makes it very confusing as we have many different competing ideas.

I think we need to get the core API solid then build on it to get to the blending features. For that reason I sadly suggest that we reject this PR and revisit the concept once the lowest level API is in place.

I don't think this API is sufficient for all needs, but it is a good start, but I don't want to lock us into this simplistic API and bypass other use cases that may need similar if not identical code. Blending and IK/procedural animation are very similar, they are all ways to manipulate a skeleton in a relative manner, and I think we need a core way to do that first, then implement blending on top of it.

JeffM2501 avatar Jan 16 '25 16:01 JeffM2501