Supporting membership autorenewal

[omarabuhussein]

Overview

Adding support to allow users to autrenew their memberships when they pay using any offline payment method.

Note that this PR replaces part of what this PR does : https://github.com/colemanw/webform_civicrm/pull/181 in order to make changes as little as possible in each PR.

Before

"Webform civicrm" have a limited support for autorenew which require you :

1- frequency unit field to be set. 2- frequency interval field to be set. 3- Number of installments > 1

If the 3 conditions above are met and there is a membership configured on the webform then the module will assume by default that it is an autorenew membership which is not necessarily the case. for example you might need to pay for certain membership in monthly installments but at the same time you don't it to be autorenewed. or maybe the user want to autorenew the membership but in one installment at each renewal.

After

A new field that allow you or the user to select if the membership to be autorenewed or not, To configure a membership to be an autorenew you will have to :

1- Have this field selected and set to Yes. 2- A frequency interval and unit fields to be set. 3- the membership is paid using an offline payment processor (either "manual payment" or any payment processor that implements "Payment_Manual" core class).

if the conditions above are met then the membership will be autorenewed.

Hence again; This PR add support for for offline payment processors which are as described above (either "manual payment" or any payment processor that implements "Payment_Manual" core class), I avoided adding support to Live payment processors to make the PR simpler for review but if all agree I will add support for Live processors in another PR.

Technical Details

1- The first commit https://github.com/colemanw/webform_civicrm/commit/d80bce9ca7084e0ea808240f117600ea7f8f536b in this PR add the UI on the admin panel to allow admins to show or hide the "autorenew" field

2- The 2nd commit implements the logic bedhinde the autorenew, it is the minimal code needed to make this work, here is some explanation to the changes I have done :

• This part :

      if (isset($this->ent['contribution_recur'][1]['id']) && !empty($params['auto_renew'])) {
        $contributionRecurId = $this->ent['contribution_recur'][1]['id'];
        $contributionRecur = civicrm_api3('ContributionRecur', 'get', [
          'return' => 'auto_renew',
          'sequential' => 1,
          'id' => $contributionRecurId,
        ]);
        if (!empty($contributionRecur['values'][0]['auto_renew'])) {
          $params['contribution_recur_id'] = $contributionRecurId;
        }
        unset($params['auto_renew']);
      }

replaces the old code to allow only memberships configured to be autorenewed to get actually autornewed and not any membership with a recur contribution configured.

• This part :

    if ($paymentType === 'deferred' && !empty($contributionParams['auto_renew'])) {
      $contributionRecurParams['auto_renew'] = 1;
    }

Ensure that only manual payment processors payments set the auto_renew field on the recurring contribution.

• This part :

    $isThereMembershipToBeAutoRenewed = $this->isThereMembershipToBeAutoRenewed();
    if (($numInstallments != 1 || $isThereMembershipToBeAutoRenewed) && !empty($frequencyInterval) && $this->contributionIsPayLater) {
      if ($isThereMembershipToBeAutoRenewed) {
        $params['auto_renew'] = 1;
      }

  private function isThereMembershipToBeAutoRenewed() {
    $autoRenewMembership = FALSE;
    foreach ($this->data['membership'] as $contactMemberships) {
      foreach($contactMemberships['membership'] as $membership) {
        if (!empty($membership['auto_renew'])) {
          $autoRenewMembership = TRUE;
        }
      }
    }
    return $autoRenewMembership;
  }

Set the "auto_renew" on the recur contribution if there is any membership configured to be autorenewed.

[omarabuhussein]

@KarinG

1- As described in the PR description, this unlike my previous PR only touch the offline payments and do the minimum stuff to make the autorenewal thing work.

2- Now not sure if you are okay with that but I am thinking to create another PR for Live payment processors, but maybe I need to think of that before doing anything in case some Live payment processors implementations already handle autorenewals by themselves.

3- I did my tests locally for this but I believe you will probably need Live tests, so while the PR is ready for review it is not ready for merge yet even if it pass your review and get approved, that will be the case until I can get this PR into one of our clients Live sites and have it been tested for sometime there.

[jamienovick]

Hi @KarinG - trying to find a better way to get this in. Hope this is going in the right direction. Do let us know your thoughts! Thanks

[KarinG]

Thank you - I’ll have another look at it.

[omarabuhussein]

@mattwire

I really don't think we should be treating "manual" and payment processor based renewals any differently. We don't in CiviCRM core! You should just pass the params to the payment processor and let it do it's thing.

Neither do I, my initial PR already handled both because I feel that they belong together, but it was @KarinG request to split the PR to make them shorter to make testing and review easier

[KarinG]

No - that's not what I suggested. I didn't see the need to refactor existing code generating recurring contributions - in order to achieve supporting membership autorenewal. I did not suggest that you split up manual and payment processor methods and agree with @mattwire that they should not be handled differently.

[omarabuhussein]

No - that's not what I suggested. I didn't see the need to refactor existing code generating recurring contributions - in order to achieve supporting membership autorenewal. I did not suggest that you split up manual and payment processor methods and agree with @mattwire that they should not be handled differently.

Then I misunderstood what you meant by

As we’ve discussed in the past we really need PRs to just do one thing so that we can test it.

in the previous PR

I thought you also had an issue with the previous PR touching the live payment processors, that's why I did not include it.

Anyway I re-added the live payment support for this PR.

[mattwire]

@omarabuhussein Looks like this is almost there :-) I've commented above. It is a shame that the Order API in CiviCRM does not yet support recurring contributions as it would be nice to change to that and not have to discuss where we should be saving IDs etc..

[omarabuhussein]

thanks @mattwire :-)

[KarinG]

1. I want to go back to the initial use case for the implementation of paying in installments - this was based on a feature that a client who had previously worked with MS Dynamics had requested: for Donors to select an amount and then indicate whether or not they wanted to pay that now or in installments.

Example: $1200 fees -> select 1) pay now or 2) pay in installments and frequency (configurable, e.g. 12, and monthly):

selecting 1) pay now it just becomes a one-time contribution

selecting 2) pay in installments it charges $100 upon webform submission + sets up a recurring series in CiviCRM: 12 installments; monthly; $100; next scheduled date: one month from now;

2. This also then just worked for Event fees as webform does not care what the monies are for -> so e.g. $1800 in swim fees - a swim season is typically 9-10months or day-care fees for the entire year (Sep-June); -> Fees (any money) can be configured to be 1) pay now; or 2) pay in installments (e.g. $200 every month for 9 months);

3. And it also just works for Membership You could decide on a specific number of renewals (commitment) and then that # of terms; e.g. 3 (for a 3x 1year membership); What that does (before this PR) is:

Example: 3 x $10 (Membership $10/y) + 5% tax = $31.50 Top up Contribution = $10 Total: $41.50 Number of Installments: 12

This correctly produces: a first time charge of $1.25 + a recurring series for the amount: $1.15 for a total of 36 installments - so $1.25 + 11 x $1.15 = $41.50

4. In this PR - are we trying to make it such that a Membership can be configured in CiviCRM Admin for a 1 month duration - and that after every (recurring) contribution that is linked to that Membership -> the Membership is renewed for another month? I know that this is how admins in CiviCRM native have gone about simulating recurring Membership but wasn't that a workaround because you really just want to be able to pay a Membership in 12 installments (e.g.)?

Having a 1y Membership paid in 12 installments - configured to autorenew - are we not going to end up with 12y of Membership? Or should it be such that the autorenew field can only be checked if and when number of installments = 1? In that case how would you pay for a e.g. Dental Technologist Membership ($865) in monthly installments and have it renewed annually?

[KarinG]

PS - if in CiviCRM Core you purchase a Membership that is Recurring (autorenew) -> it sets up a Recurring Series with unlimited installments (i.e. 0);

[jamienovick]

Hey Karin,

CiviCRM core in general does not actually have anything specific to support “auto-renewal”. The auto-renewal processing is the job of the payment processor, and all CiviCRM provides is the means to facilitate that auto-renewal by having the following:

1- The Recur Contribution entity. 2- The Recur contribution ID field on the Membership entity (contribution_recur_id)

But other than the above and some UI elements that CiviCRM adds, it is the responsibility of the payment processor implementation to provide the actual “auto-renewal”.

So to understand better what this PR does, let's take an example on how auto-renewal work currently in CiviCRM core with any payment processor that supports auto-renewal:

1- The membership type need to be configured to support auto-renew 2- A contribution page need to be created with the membership type enabled and auto-renew enabled (there is also a setting to give the user the option to auto-renew or not) 3- The user access the public contribution page and sign up for the membership. 4- once the user go through the contribution page pages and submit the form a new membership will be created as well as a recur contribution and the contribution_recur_id field on the crated membership will be set and points to the newly created recur contribution.

That's it, at this point CiviCRM’s job ends and the membership will appear as auto-renew on the memberships tab. It is then the payment processor that is used to do the actual renewal. And by actual renewal I mean such things as extending the membership end date and creating the new contribution or any other functions you want the payment processor to perform.

CiviCRM core job in this case was only to provide the UI during configuring the contribution page and allowing auto-renewal to be configured, as well as creating the recur contribution and setting the contribution_recur_id field on the membership.

In this PR we are just doing exactly the same as how you would expect to configure a membership on the contribution page to auto-renew by default (or giving the user the option to auto-renew or not). The field added also just does that, and similar to how CiviCRM will create the recur contribution and set the contribution_recur_id on the membership, I also set them here in this PR.

The rest other details you are talking about such as extending the membership end date, creating further contribution on auto-renewal are the job of the payment processor that is used during configuring the webform and not affected here.

In short, what this PR adds is just something similar to the job of this field :

Hope that is clear, let us know if you need any further details.

Jamie and Omar

[KarinG]

"It is then the payment processor that is used to do the actual renewal. And by actual renewal I mean such things as extending the membership end date"

No, no - you've got it wrong. Payment processors are not responsible for the business logic (like extending a Membership end date). Payment processors don't care what the monies are for. The business logic of what happens with money connected to an Event vs to a Membership resides within CiviCRM (and should stay there).

In case of recurring contributions one would use repeattransaction API to create the next Contribution in a recurring series. That copies many (all) of the parameters of a previous Contribution in a recurring series into a new Contribution with Status Pending (a Core workflow).

The payment processor code's job is to subsequently attempt to secure a Payment -> and if successful (Payment processor has a result) one would typically use completetransaction API to Complete that Pending Contribution. It is during that API call that CiviCRM Core handles the business logic of what should happen as a result of having successfully (or not) transacted. Note that instead of live attempting to transact, a processor could also load results from a CSV e.g. to Contributions with Pending status and then run completetransaction API after that.

A Membership configured as Recurring in Membership Type settings and on a form allowed as Recurring i) creates a Recurring Series with an open ended (0) installments, ii) with the Duration set in the Membership Type config and iii) with Membership -> linked to the Membership Type. Example screenshot below (this is 5.19.x):

So again - note: Installments: 0 Frequency: every 1 day (result of Membership Type configuration) Auto Renew is set to Yes There was a Contribution with successful Payment Today -> Nov. 15 The Next Contribution is scheduled for Nov. 16, because that's Today + 1 day (Frequency)

Tomorrow morning a new Contribution will be created with status Pending. A Payment will be attempted. And if we get a result from the Payment processor completecontribution API will update the Contribution to either Completed or Failed. And if Completed then CiviCRM Core will handle the logic of extending the Membership End date by +1 day in this case. And if Failed then it will not and will set the Membership Status to Grace (or whatever the Status rules are).

Does this make sense now?

[jamienovick]

Hi @KarinG sorry for slow reply. Lots of level of review in the big 'ol compucorp machine

Lets agree to disagree on the point above as it's not really the purpose of this PR. we can have that debate another time over some warm beer!

The purpose of this PR, is to set the membership contribution_recur_id field and optionally the auto_renew field on the recur contribution entity. Thats all.

It's really essential to have these so that any "manual" or offline payment processor then knows whether to renew a membership. Without it there's no way to set that when someone signs up from webform.

Can you let us know if this PR is acceptable or if not whether it needs to be reworked and how? We are simply adding the same functionality to webform that exists on contribution pages so hopefully its on the right lines.

Please let us know if you want to discuss on a call or on chat as this has been going on for a while now (years since the original PR!) and it's really wasting everyones time. Time to wrap it up!

Many thanks

Jamie + Omar

[KarinG]

I am sorry you feel that I'm wasting people's time but I'm not comfortable with these changes and remain very concerned about the ongoing misinterpretations of existing functionality. The comment in the Before -> clearly shows a lack of understanding how it works at the moment for clients who are using the existing webform CiviCRM functionality to pay for Memberships in instalments.

I apologize if I'm not explaining this well - I'm going to try once more:

This you can do currently (and there is no assumption in the module that Membership is configured to autorenew): Membership A (1y) -> $120 -> $10/month -> recurring series: 12 instalments; monthly $10/each.

This you can also do: Membership Terms -> 2 Membership A (2y) -> $240 -> $10/month -> recurring series 24 instalments; monthly $10/each.

If Membership A (1y) is configured to be AutoRenew -> what should happen after the 12 instalments have completed? It's another dimension on the Recurring Series -> it's a Recurring Recurring Series. One option to flatten that could be that the existing Recurring series would get another 12 instalments -> from 12 -> to 24.

Membership Recurring in CiviCRM Core means instalments = 0 (open ended). That's effectively not unlike bumping up Membership Terms and creating e.g. a recurring series with 48 or 60 instalments; monthly $10/each. In both cases you're taking $10/month for a very long time.

And instalment = 1 bypasses the entire recurring bits and will process the Membership as it always has.

[mattwire]

Membership Recurring in CiviCRM Core means instalments = 0 (open ended). That's effectively not unlike bumping up Membership Terms and creating e.g. a recurring series with 48 or 60 instalments; monthly $10/each. In both cases you're taking $10/month for a very long time.

And instalment = 1 bypasses the entire recurring bits and will process the Membership as it always has.

@KarinG That's not quite right. CiviCRM doesn't enforce much at all based on the values of the database fields, it's pretty much up to the payment processor how they handle them.

The relevant fields are:

• civicrm_membership.contribution_recur_id - Conditional foreign key to civicrm_contribution_recur id. Each membership in connection with a recurring contribution carries a foreign key to the recurring contribution record. This assumes we can track these processor initiated events.

• civicrm_contribution_recur.installments - Total number of payments to be made. Set this to 0 if this is an open-ended commitment i.e. no set end date. In practise this has no meaning within CiviCRM core, but IS used in slightly different ways by various payment processors.

• civicrm_contribution_recur.auto_renew - Some systems allow contributor to set a number of installments - but then auto-renew the subscription or commitment if they do not cancel. - Not used at all by any payment processors that I'm involved in, but I guess is used by eg. iATS and I can see why you would want to.

• Installments can already be set by webform_civicrm and is not being touched by this PR.

• The auto_renew flag can be set or not when creating recurring contributions via the API so I don't see why this could not be exposed as an option to set via webform_civicrm - it will be used by different payment processors in different ways and not at all by some.

• The contribution_recur_id field should, in my opinion, always be set if the contribution_recur record is being used to pay for that membership. That's what the UI in CiviCRM expects and it will be impossible to track payments against memberships without this unless you implement something custom that's not part of CiviCRM core.

[KarinG]

You guess is incorrect re: iATS. And I'm pretty sure I understand the Webform CiviCRM side of instalments here.

Can you answer this question: This is 100% a Webform CiviCRM question as you can not configure this scenario in Core at the moment: If Membership A (1y) is configured to be AutoRenew -> what should happen after the 12 instalments have completed? It's another dimension on the Recurring Series -> it's a Recurring Recurring Series. One option to flatten that could be that the existing Recurring series would get another 12 instalments -> from 12 -> to 24. In other words -> we're really looking at how do we create a Renewal of a Recurring Series (of instalments).

This scenario does not exist in CiviCRM Core b/c you can not pay for an annual Membership in instalments. The recurring series that is created when you do this in Core has frequency: year.

I'm just not risking breaking changes to existing sites using Webform CiviCRM to have clients pay for Membership (and Events) in instalments. It would be easy to end up with 12y of Membership e.g.

If you guys want this feature you can sponsor it - and I'll try find time for it. I just can't seem to be able to explain this.

[KarinG]

I just talked this through with Alan and a solution would be to have an auto-renewal for Membership only if no instalments (avoiding the Recurring of the Recurring Contribution Series problem). That will give admins a Core like auto-renewal feature for Membership while still preserving the existing webform CiviCRM native concept of paying for a Membership (or Event or anything else) in instalments.

[aydun]

@KarinG 's Recurring Recurring Series and Renewal of a Recurring Series is very relevant for a client I'm working with now that maybe illustrates the difference between a 'renewal' of a recurring series for membership vs creating it initially for multiple years.

At renewal time, they want to change the price to the current membership fee. In some situations they also want to change the membership type (according to a specified progression). They also want to email them info at the start of that new membership year. If there are no changes then the renewal just sets up a further series of 12 payments, with much the same effect as an initial multi-year, but for more complex situations, the renewal produces significant changes.

[KarinG]

@aydun - right now you would need to increase the $amount of the recurring series. It’s what xero just did to me! I pay monthly and they just provided notice that they are adding a few more $ to my monthly payment.

I’m happy you understand what I mean by Recurring Recurring Series. I have one client for which I can look at that too. But first let me add in Membership AutoRenew for the pay all in one go (i.e. no recurring series dimensions). I can definitively use some bits from this PR for that.

[omarabuhussein]

No, no - you've got it wrong. Payment processors are not responsible for the business logic (like extending a Membership end date). Payment processors don't care what the monies are for. The business logic of what happens with money connected to an Event vs to a Membership resides within CiviCRM (and should stay there). In case of recurring contributions one would use repeattransaction API to create the next Contribution in a recurring series. That copies many (all) of the parameters of a previous Contribution in a recurring series into a new Contribution with Status Pending (a Core workflow).

I think I disagree with you here Karin , at that end isn't it the payment processor IPN class what calls the repeattransaction API call ? in other words if I am now to create a civicrm extension/implementation for any payment processor out there that support recur series, then it is up to me as the payment processor implementer to decide if I want to call repeattransaction or not, which means it is actually the payment processer who implements the business logic and it happened that many payment processors inside CiviCMR just happened to use repeattransaction core API because it exactly do what they want and it save them time from writing all of the functionally inside it from scratch.

The payment processor code's job is to subsequently attempt to secure a Payment -> and if successful (Payment processor has a result) one would typically use completetransaction API to Complete that Pending Contribution.

I think this also proves my point above, because when you say "one would typically completetransaction", "one" means here the IPN class or the payment processor extensions and "typically" means I have the power inside my payment processor implementation not to call it and do whatever I want to do.

Tomorrow morning a new Contribution will be created with status Pending. A Payment will be attempted. And if we get a result from the Payment processor completecontribution API will update the Contribution to either Completed or Failed. And if Completed then CiviCRM Core will handle the logic of extending the Membership End date by +1 day in this case. And if Failed then it will not and will set the Membership Status to Grace (or whatever the Status rules are).

That totally correct as long as the payment processor itself supports recur series and it is configured to call the payment processor extension IPN class.

This you can do currently (and there is no assumption in the module that Membership is configured to autorenew): Membership A (1y) -> $120 -> $10/month -> recurring series: 12 instalments; monthly $10/each. This you can also do: Membership Terms -> 2 Membership A (2y) -> $240 -> $10/month -> recurring series 24 instalments; monthly $10/each.

correct, but again, only the first installment will be created here (10$) as well as recur contribution, the remaining contributions will be created only if the payment processor supports recur series and is configured correctly.

If Membership A (1y) is configured to be AutoRenew -> what should happen after the 12 instalments have completed? It's another dimension on the Recurring Series -> it's a Recurring Recurring Series. One option to flatten that could be that the existing Recurring series would get another 12 instalments -> from 12 -> to 24.

Yes we can call it "Recurring recurring series", but what would happen here is up to the payment processor extension, in our case inside membershipextras at the time of renewal we create another 12 installments (so 24 installments total) and so on every year/term.

Can you answer this question: This is 100% a Webform CiviCRM question as you can not configure this scenario in Core at the moment: If Membership A (1y) is configured to be AutoRenew -> what should happen after the 12 instalments have completed?

you cannot do this at webform module now too either since it does not support the auto-renew flag (which is basically the civicrm_membership.contribution_recur_id field).

it's a Recurring Recurring Series. One option to flatten that could be that the existing Recurring series would get another 12 instalments -> from 12 -> to 24. In other words -> we're really looking at how do we create a Renewal of a Recurring Series (of instalments).

Yes and as I explained above it is up to the payment processor extension to decide what to do here, and as I said before we actually do this inside membershipextras extension, but other payment processor extensions may decide to do something else, CiviCRM core at the end does not enforce anything here.

I'm just not risking breaking changes to existing sites using Webform CiviCRM to have clients pay for Membership (and Events) in installments. It would be easy to end up with 12y of Membership e.g.

This change here should not affect any existing client since CiviCRM does not do anything about it and since the auto-renewal is not supported at the moment by the webform either, it should only affect who enable the auto-renewal option inside the webform in case they are also using a payment processor which support auto-renewal.

But anyway I can really understand your concerns here Karin, It just seems that we have a bit different opinions and understanding about how all of this currently work and how it should work.