Feature request: add TextDecoder and TextEncoder functions

[Shonke]

Feature Description

I need to convert between ArrayBuffer and String. I haven't found an efficient conversion method yet I have to use encodeURI and decodeURI:

function str2buf (str) {
  if (typeof str !== 'string') return EMPTY_BUF
  const arr = []
  const a = encodeURI(str)
  for (let i = 0; i < a.length;) {
    if (a[i] === '%') {
      arr.push(Number.parseInt(a.slice(i+1, i+3), 16))
      i += 3
    } else {
      arr.push(a.charCodeAt(i))
      i++
    }
  }
  const buf = Uint8Array.from(arr)
  return buf.buffer
}
function buf2str(buf) {
  if (!buf) return ''
  if (Object.prototype.toString.apply(buf) === '[object ArrayBuffer]') buf = new Uint8Array(buf)
  if (Object.prototype.toString.apply(buf) !== '[object Uint8Array]') return ''
  const length = buf.byteLength
  const arr = []
  for (let i = 0; i < length;) {
    // ascii
    if (buf[i] < 0x7f) {
      arr.push(String.fromCharCode(buf[i]))
      i++
    } else {
      const cc = []
      for (;i > length || (buf[i] > 0x7f);) {
        cc.push('%')
        cc.push((buf[i]).toString(16).padStart(2, '0'))
        i += 1
      }
      arr.push(decodeURI(cc.join('')))
    }
  }
  return arr.join('')
}

Suggested Solution (optional)

No response

Already existing or connected issues / PRs (optional)

No response

[na--]

Thanks for opening this issue! Indeed, the Encoding API seems like a useful one to have in k6. Though it is a Web API, not an ECMAScript one, so I am not sure if our JS runtime, goja, would be open to merging it or not :thinking: If it isn't suitable for there, we could potentially merge it into just k6, though I am not promising anything just yet.

A good first step here would be creating a well-tested JavaScript xk6 extension with the functionality. This will solve the immediate problem and greatly simplify its potential inclusion in k6.

[Shonke]

I think it's a good idea to put it into xk6 extension

[arukiidou]

What about importing pako as a remote fslib? This could work for free (without using xk6) because neither esm nor node

https://gist.githubusercontent.com/Yaffle/5458286/raw/1aa5caa5cdd9938fe0fe202357db6c6b33af24f4/TextEncoderTextDecoder.js

[mstoykov]

Just passing by to leave an example using a much fuller polyfill

import { TextEncoder } from "https://raw.githubusercontent.com/inexorabletash/text-encoding/master/index.js"

let te = new TextEncoder()

console.log(te.encode("someString").byteLength)

console.log(te.encode("Нещо на Български").byteLength)

export default function() {} // just to not make k6 error

[toralux]

I have problems using this npm package https://www.npmjs.com/package/jose with k6 because it require TextDecoder and TextEncoder functions. I use it to log-in and get JWT token to use for my REST-api tests that require authentication. Ideally I would like TextDecoder and TextEncoder to be supported in k6 though...

@mstoykov Will your workaround work with 3rd party libraries that need TextDecoder and TextEncoder as well?

[mstoykov]

I have no idea @toralux as I have no idea what else jose uses in order to know if this will be enough of a fix.

if you have file polyfill-text.js:

import { TextEncoder,TextDecoder } from "https://raw.githubusercontent.com/inexorabletash/text-encoding/master/index.js"
globalThis.TextEncoder = TextEncoder;
globalThis.TextDecoder = TextDecoder;

And you import it before whatever needs TextEncoder - it will be there for the dependancy

import "./polyfill-text.js"; // file from above
import jose from "./path/to/jose"; // thing needing TextEncoder
// rest of code

To what extend that will help you again depends on what else jose or w/e needs to actually function.

[oleiade]

@toralux 👋🏻 you might also consider my xk6-encoding extension, which provides the TextEncoder and TextDecoder constructs to k6. You would need to import them still, which might get in the way of solving your issue though 🙇🏻