Upgrade bid adjustments feature

[bretg]

We discussed in last week's PBS committee meeting that it would be good to have the option for Prebid Server host companies to be able to globally deduct a hosting fee from all bids to reduce the discrepancy caused in publisher reporting by not having true net-bids in this scenario. In subsequent meetings we've agreed to expand and generalize the existing bid adjustments feature.

The proposed configuration is hard-coded to support only 3 dimensions: mediatype, bidder, and deal. This config could come in on the request or be present in account config.

Here's an example on the request

{
  "ext": {
    "prebid": {
      "bidadjustments": {         // new syntax
          "mediatype": {
              "banner": { // values can be: banner, video-instream, video-oustream, native, audio, or *
                  "bidderA": {
                      "*" : [ {"adjtype": "multiplier", "value": -0.1} ]
                  },
                  "*": {  // wildcard bidder only matches if there isn't a more direct match
                      "111111": [{  // specific deal
                          "adjtype": "static", "value": 3.00, "currency": "USD"
                      }],
                      "*": [{  // other deals
                            "adjtype": "cpm": -0.1, "currency": "USD"
                      }]
                  }
              },
              "video-instream": {
                  "*": { // all bidders
                     "*": [{ // all deals
                         "adjtype": "multiplier", "value": 0.90
                     }, {
                         "adjtype": "cpm", "value": 0.18
                     }]
                  }
              }
          }
       }
    }
  }
}

Note that the last example in this model supports advanced adjustment scenarios like multiply the bid by 0.9 and then subtract 0.18. The adjustments would be applied in the order defined in the array.

Any module that needs to reverse the adjustment (e.g. floors), will need a way to get the relevant adjustment array and they can walk the adjustments backwards. This implies that there should be a function similar to the Floors getFloor() function that allows callers to supply dimension values and get the adjustments array.

Prerequisite: We need to make sure the seatbid.bid.ext.origbidcpm/origbidcur feature is in place and working. Analytics adapters need to have access to both the original and adjusted bid prices.

This all has to work in conjunction with the existing

Proposed algorithm:

0. Merge request-level ext.prebid.bidadjustments with account-level bidadjustments in the normal way: deep merge with preference for the request-level with array overwrite.
1. If an adjtype configuration is specified, validate it.
   1. adjtype must be cpm, multiplier, or static
   2. if cpm, value must be >= 0 and < MAXINT
   3. if cpm, currency must be specified
   4. if multiplier, value must be >=0 and < 100
   5. if static, value must be >= 0 and < MAXINT
   6. if static, currency must be specified
   7. if "multiplier", currency is ignored
2. If validation fails:
   1. ignore bidadjustments for this request.
   2. add a warning when in debug mode
   3. log an error at N% sampling
3. Before any changes are made, store the original values in seatbid.bid.ext.origbidcpm/origbidcur
4. For each bid response, perform the existing bidadjustmentfactors first.
5. For each bid response, immediately after any bidadjustmentfactors, process this new bidadjustments structure to find an array of adjustments. If multiple adjustment arrays match, choose the one that has the least number of wildcards in the path. For each entry in the adjustment array:
   1. if adjtype is specified and "multiplier", update the bid response price multiplying by the value. e.g. if the bid price is $2.00, and the multipiler is 0.99, the adjusted price would be 2.00*0.99=1.98. Round to 4 decimal places.
   2. if adjtype is specified and "cpm", the first task is to convert the adjustment to the bid currency. Then subtract that value from the price. e.g if the bid price is $2.00 USD, and the cpm is 0.01 EUR, the adjusted price would be $2.00- $0.011=1.989. Round to 4 decimal places.
   3. if adjtype is "static", overwrite the bid response price to the currency-adjusted static value and the currency to the supplied static currency.

[bretg]

We discussed in PBS committee last week and agreed to take another crack at this feature integrated as part of the existing bidadjustmentfactors feature.

The additional requirements are:

• support an "all bidders" wildcard
• allow CPM as well as percentage adjustments, possibly different per deal
• support account-level config in addition to request-level config
• support completely static bid values for a particular deal ID

Looking at the existing bidadjustmentfactors feature, there are couple of issues with how it evolved -- I take the blame. So I propose a revised version that fixes these issues:

{
  "ext": {
    "prebid": {
      "bidadjustments": {         // new syntax
          "mediatypes": {
              "*": { // values can be: banner, video-instream, video-oustream, native, audio, or *
                  "bidderA": {
                      "percent": 0.9
                  },
                  "*": {  // wildcard bidder only matches if there isn't a more direct match
                      "cpm": {
                          "value": 0.01,
                          "currency": "USD"
                      }
                  }
              }
          }
       }
    }
  }
}

So the account level config could look like:

auction:
    bidadjustments:
         JSON-structure-matching-ext.prebid.bidadjustments above

[bretg]

Discussed in last week's PBS meeting. Additional requirements were added that requires some re-thinking:

• adjustments may differ per deal ID
• adjustments can be static, i.e. not relative to the bidresponse value

This is starting to feel a lot like the level of flexibility that exists in the Price Floors module. So here's a cut at modeling the config following a similar syntax as https://docs.prebid.org/dev-docs/modules/floors.html#schema-2

{
  "ext": {
    "prebid": {
      "bidadjustments": {         // new syntax
          "schema": {
              "fields": [ "mediatype", "bidder", "deal" ]
          },
          "values": {
              "banner|bidderA|*": { "adjtype": "multiplier", "value": 0.90 },
              "banner|*|111111": { "adjtype": "static", "value": 3.00, "currency": "USD" },
              "banner|*|*": { "adjtype": "cpm", "value": -0.01, "currency": "USD" }
          }

Runtime behavior would be similar to the enforcement phase of the floors module: loop through the values (aka rules), match the first one, apply the results to the bidresponse.

If this general approach seems viable, I'll come up with some more examples and a few more details about edge cases.

[patmmccann]

@khatibda for visibility

[patmmccann]

This proposal looks really good to me, would PBS be able to understand the old and new formats simultaneously? Would publishers doing just bidder adjustments need to change anything?

[bretg]

@patmmccann - it would be nice to simplify the feature and say that if bidadjustments are specified, then bidadjustmentfactors is ignored. But open to the behavior here -- which would take precedence? We could probably translate the old syntax to the new but need to know the precedence.

[bretg]

I've been asked to defend the perceived complexity of the "schema/value" approach. The alternate approach would be to support a full set of permutations in strict JSON. We would have to pick a solid order like "mediatype", "bidder", "deal". Here's an example of what a 3-level json could look like:

{
  "ext": {
    "prebid": {
      "bidadjustments": {         // new syntax
          "version": 1,  // needed in case we extend to additional attributes someday
          "mediatypes": {
              "banner": { // values can be: banner, video-instream, video-oustream, native, audio, or *
                  "bidderA": {
                      "*" : {  // all deals
                          "adjtype": "percent", "value": -0.1
                       }
                  },
                  "*": {  // wildcard bidder only matches if there isn't a more direct match
                      "111111": {  // specific deal
                          "adjtype": "static", "value": 3.00, "currency": "USD"
                      },
                      "*": {
                           { "adjtype": "static", "cpm": -0.1, "currency": "USD" }
                      }
                  }
              },
              "video-instream": {
                  "*": {
                     "*": {
                         { "adjtype": "percent", "value": -0.1 }
                     }
                  }
              }
          }
       }
    }
  }
}

So quite a bit wordier in the protocol, but the worst part would be if an extension was ever needed, it would explode the JSON into even finer-grained bits. And adding new dimensions isn't unlikely... adjustments for bandwidth, deviceType, or country could be added in the future. The "schema" model handles extensions cleanly.

[bretg]

I propose that if bidadjustments is specified, then bidadjustmentfactors is ignored. Will give this one more run by the committee in the next meeting.

[bjorn-lw]

Has there been made any progress for PBS-Go when it comes to bid floors and bid adjustments?

We currently have a pretty urgent issue with one publisher who has high bid adjustment factors for certain bidders (resellers that take a large cut of the revenue). Even if the SSPs bid net it doesn't really help in cases like this as the resellers cut is not accounted for.

So as it works now we specify, for example, a bid adjustment of let's say 0.7, but the floor price will not be recalculated as is done on the client side using the floor price module, meaning the S2S SSPs never gets to know the real floor price it has to meet, resulting in a large number of dropped and wasted bids.

We thought of doing some kind of interim solution in a fork of PBS and just adjust the floor per bidder whenever there is a bid factor set but it will likely be a maintenance nightmare. We probably don't want to make it into the official branch either because it's not really the solution we would want in the end (the proper solution would be to use the floor price module on the client and send the exact same setup to PBS I would assume).

Sorry for the rambling, perhaps I'm not even in the correct issue to write this, but I can't be in that wrong place (I hope).

Any comments?

[bretg]

@bjorn-lw - @PubMatic-OpenWrap has stepped up to start the porting of the floors feature to PBS-Go in https://github.com/prebid/prebid-server/pull/2341

Not sure if that PR covers updating the floors for bid adjustments, and the PR has been sitting for bit, but that's the current status

[bretg]

@patmmccann - we discussed in backlog grooming today and agreed:

• if the dimensions that could affect bid adjustments are limited to mediatype, bidder, and deal, then we'll go with a static JSON format.
• if, however, other dimensions might be wanted someday, a more flexible schema is desirable. i.e. can you envision needing to adjust the bid by domain, country, device, or size?

[bretg]

Based on issues found in PBJS, we're going to want test cases here covering the adjustment to floors.

@patmmccann also has a use case that wouldn't be covered by how this feature is speced, which is an adjustment that combines both multiplication and addition/subtraction, e.g.

updated_bid=orig_bid*factor1 - factor2

[bretg]

Based on the discussion over in PBJS, I've updated the proposal here to support an array of adjustments. I think this will work for the new use case of

updated_bid=orig_bid*factor1 - factor2

e.g. the array of adjustments would be

[{ "adjtype": "multiplier", "value": 0.90 }, { "adjtype": "cpm", "value": 0.18 }]

So a bid of 1.00 would get adjusted this way:

1. 1.00 * 0.90 = 0.90
2. 0.90 - 0.18 = 0.72

The floors system would invert the order of the adjustments as well as the operators. e.g. if the floor is 1.0, this is what would be sent to the bidder:

1. 1.00 + 0.18 = 1.18
2. 1.18 / 0.9 = 1.32 (rounded up from 1.31)

So if the bidder bids their floor of 1.32, it's adjusted to a bid value of 1.32*0.9-0.18=1.008

[patmmccann]

This appears to cover our case perfectly! Thank you

[bretg]

Ok, we're locking the protocol into exactly 3 dimensions: mediatype, bidder, and deal. Updated the description's example.

[bsardo]

@bretg we’re about to begin work on this for PBS-Go but I just wanted to clarify a few things first.

1. As you mentioned in the description, we should process bidadjustmentfactors first and then bidadjustments right? My initial thought was bidadjustmentfactors would be ignored if bidadjustments was present but it sounds like they work in conjunction.
2. I assume we’re merging the request-level and account-level configs via a standard JSON merge right?
3. I see we’re supporting three dimensions. Is mediaType always the first dimension, bidder the second and deal the third or can they be arranged in any order?
4. Is the publisher required to specify all three dimensions (vs just specifying the first dimension or first two) for the config to be considered valid?
5. I assume the wildcard is simply a catch-all but need not be specified right? The order in which dimension values are specified does not matter.
6. One of the requirements in the description reads: if adjtype is "static", simply overwrite the bid response price to the currency-adjusted static value and the currency to the supplied static currency. which contradicts another requirement in the description that states: if static or multiplier, currency is ignored. Is the supplied static currency supposed to overwrite the currency or are we supposed to ignore the static currency?

Thanks in advance.

[bretg]

1. yes.
2. Added Merge request-level ext.prebid.bidadjustments with account-level bidadjustments in the normal way: deep merge with preference for the request-level with array overwrite.
3. Always mediaType, bidder, and deal in that order
4. All 3 dimensions must be specified
5. Don't understand. Something's got to be there, right?
6. Good catch, fixed. We wound up wanting to support currency for static.

[bretg]

Brian and I discussed

I assume the wildcard is simply a catch-all but need not be specified right? The order in which dimension values are specified does not matter.

Updated the description to include "If multiple adjustment arrays match, choose the one that has the least number of wildcards in the path. "

[bsardo]

@bretg I see the following comment in the code snippets here: // values can be: banner, video-instream, video-outstream, native, audio, or *. The actual code snippets though show video as one of the values. What are the actual video related media types that we should support? Is it just video?

[bretg]

@bsardo - fixed! We can make the config be explicit: video-instream and video-outstream.

[AlexBVolcy]

Upgrade Bid Adjustments was developed and merged. Here is the merged PR: https://github.com/prebid/prebid-server/pull/2678