mcrouter icon indicating copy to clipboard operation
mcrouter copied to clipboard
verified publisher icon facebook

Accessing memcached servers via routes

Open chilumbugeorge opened this issue 7 years ago • 12 comments

We have two sets or pools of memcached, and would like to use mcrouter for scaling our memcached deployments as well as provide us with HA protection. Memcached hosts in each pool should have the same data.

I have setup 2 mcrouters to perform routes as well as HA protection just for testing purposes. The mcrouter config file on each mcrouter server looks like this:

 {
  "pools": {
    "memcached-event": {
      "servers": [
        "10.1.9.100:11211",
        "10.1.9.101:11211"
      ]
    },
    "memcached-notification": {
      "servers": [
        "10.1.9.200:11211",
        "10.1.9.201:11211"
      ]
    },
"route": {
     "type": "OperationSelectorRoute",
     "operation_policies": {
       //***routes for hosts in the event pool***
       "add": "AllSyncRoute|Pool|memcached-event",
       "delete": "AllSyncRoute|Pool|memcached-event",
       "get": "LatestRoute|Pool|memcached-event",
       "set": "AllSyncRoute|Pool|memcached-event"
       //***routes for hosts in the notification pool***
       "add": "AllSyncRoute|Pool|memcached-notification",
       "delete": "AllSyncRoute|Pool|memcached-notification",
       "get": "LatestRoute|Pool|memcached-notification",
       "set": "AllSyncRoute|Pool|memcached-notification"
   }
}

The mcouters are both up and running, and can telnet to a memcached host from an mcrouter server with a command like this for example:

telnet 10.1.9.100 11211 However, i would like to know 2 things:

With my config above, how can i access a memcached server using the mcrouter routes? I am basically trying to see an example of how routes in mcrouter are used to access memcached.

How can i check that my mcrouters are working as a cluster? How can i check that they are communicating?

chilumbugeorge avatar Feb 06 '18 04:02 chilumbugeorge

Hi George,

First, make sure that you're running mcrouter with the argument --config=file:/path/to/your/config.json If you do that with the config file you posted, mcrouter will fail to load because of malformed json and a missing "d" in the pool names. Once you fix that, you can run memcache commands to test your setup: printf "set abc 0 0 4\r\n1234\r\n" | nc localhost 5000 printf "get abc\r\n" | nc localhost 5000

That is assuming you picked 5000 as the port (-p 5000).

orishu avatar Feb 06 '18 19:02 orishu

I have fixed the typo by adding the "d" appropriately. By config=file:/path/to/your/config.json, do you mean the /etc/mcrouter.conf file coz i cannot locate any .json file associated with mcrouter?

I have started mcrouter with /etc/init.d/mcrouter start, and here is how the process looks like:

mcrouter 5618 1 0 Feb01 ? 00:01:13 /usr/bin/mcrouter --debug-fifo-root /var/lib/mcrouter/fifos --stats-root /var/lib/mcrouter/stats -L /var/log/mcrouter/mcrouter.log -p -f /etc/mcrouter.conf -f /etc/mcrouter.conf -L /var/log/mcrouter/mcrouter.log -p 11212 --send-invalid-route-to-default --file-observer-poll-period-ms=1000 --file-observer-sleep-before-update-ms=100

And when i run service mcrouter status, i get:

**mcrouter.service - Mcrouter memcached router Loaded: loaded (/lib/systemd/system/mcrouter.service; disabled; vendor preset: enabled) Active: active (running) since Thu 2018-02-01 11:42:57 CST; 5 days ago Main PID: 10026 (mcrouter) CGroup: /system.slice/mcrouter.service └─10026 /usr/bin/mcrouter --debug-fifo-root /var/lib/mcrouter/fifos --stats-root /var/lib/mcrouter/stats -L /var/log/mcrouter/mcrouter.log -p -f /etc/mcrouter.conf -f /etc/mcrouter.conf -L /var/log/mcrouter/mcrouter.log -p 1121

Feb 01 11:42:57 db-gc-2 systemd[1]: Started Mcrouter memcached router. Feb 01 11:42:57 db-gc-2 mcrouter[10026]: E0201 11:42:57.364177 10026 mcrouter_config.cpp:39] CRITICAL: Couldn't initialize from standalone flavor file /etc/mcrouter.conf Feb 01 11:42:57 db-gc-2 mcrouter[10026]: I0201 11:42:57.364261 10026 main.cpp:416] Logging to /var/log/mcrouter/mcrouter.log**

This is how i installed mcrouter:

# Installation sudo add-apt-repository ppa:gaod/mcrouter sudo apt-get update sudo apt-get install -y mcrouter

My team packaged the PPA for mcrouter for Ubuntu 16.04.

We should be using the default port as i did not change anything but the config file posted earlier. When i run the two commands below, i do not get anything (no results or errors...nothing):

printf "set abc 0 0 4\r\n1234\r\n" | nc localhost 5000 printf "get abc\r\n" | nc localhost 5000

But lets say the commands above worked, how would mcrouter know where to store and retrieve this command since we have to memcached sets or farms for different functions (memcached-event and memcached-notification? using my config above, how can i write to memcached-event using the configured routes?

chilumbugeorge avatar Feb 07 '18 03:02 chilumbugeorge

Hi Ori. I have made necessary changed to the config file by adding the missing "d". When i run using the argument --config=file:/path/to/your/config.json as suggested, this is what i get:

george.chilumbu@db-gc-2:~$ sudo service mcrouter start --config=file:/etc/mcrouter.conf george.chilumbu@db-gc-2:~$ george.chilumbu@db-gc-2:~$ george.chilumbu@db-gc-2:~$ sudo service mcrouter status ● mcrouter.service - Mcrouter memcached router Loaded: loaded (/lib/systemd/system/mcrouter.service; disabled; vendor preset: enabled) Active: active (running) since Wed 2018-02-07 11:15:31 CST; 2s ago Main PID: 18503 (mcrouter) CGroup: /system.slice/mcrouter.service └─18503 /usr/bin/mcrouter --debug-fifo-root /var/lib/mcrouter/fifos --stats-root /var/lib/mcrouter/stats -L /var/log/mcrouter/mcrouter.log -p -f /etc/mcrouter.conf -f /etc/mcrouter.conf -L /var/log/mcrouter/mcrouter.log -p 1121

Feb 07 11:15:31 db-gc-2 systemd[1]: Started Mcrouter memcached router. Feb 07 11:15:31 db-gc-2 mcrouter[18503]: E0207 11:15:31.548229 18503 mcrouter_config.cpp:39] CRITICAL: Couldn't initialize from standalone flavor file /etc/mcrouter.conf Feb 07 11:15:31 db-gc-2 mcrouter[18503]: I0207 11:15:31.548302 18503 main.cpp:416] Logging to /var/log/mcrouter/mcrouter.log

And testing with the two commands:

printf "set abc 0 0 4\r\n1234\r\n" | nc localhost 5000 printf "get abc\r\n" | nc localhost 5000

Does not give me any results or errors. Nothing.

But even though the commands above worked, i am wondering how mcrouter can know which memcached set/farm to access since there are 2 of them, memcached-event and memcached-notification?

On Wed, Feb 7, 2018 at 3:57 AM, Ori Shalev [email protected] wrote:

Hi George,

First, make sure that you're running mcrouter with the argument --config=file:/path/to/your/config.json If you do that with the config file you posted, mcrouter will fail to load because of malformed json and a missing "d" in the pool names. Once you fix that, you can run memcache commands to test your setup: printf "set abc 0 0 4\r\n1234\r\n" | nc localhost 5000 printf "get abc\r\n" | nc localhost 5000

That is assuming you picked 5000 as the port (-p 5000).

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/facebook/mcrouter/issues/229#issuecomment-363546020, or mute the thread https://github.com/notifications/unsubscribe-auth/APLx-sX2wY5P6ziRFuZgFWbJjVsy-LGNks5tSK7FgaJpZM4R6eRc .

-- Purpose is the key to success. You can easily be successful in the wrong things.

chilumbugeorge avatar Feb 07 '18 03:02 chilumbugeorge

It sounds like you want split read requests between the two pools (e.g., randomly pick one of the two pools on each request), but have writes/deletes replicate to both pools. The following config achieves that:

{
  "pools": {
    "memcached-event": {
      "servers": [
        "10.1.9.100:11211",
        "10.1.9.101:11211"
      ]
    },
    "memcached-notification": {
      "servers": [
        "10.1.9.200:11211",
        "10.1.9.201:11211"
      ]
    }
  },
  "route": {
    "type": "OperationSelectorRoute",
    "operation_policies": {
      "add": {
        "type": "AllSyncRoute",
        "children": [
          "PoolRoute|memcached-event",
          "PoolRoute|memcached-notification"
       ],
       "name": "replicated-route"
      },
      "delete": "replicated-route",
      "set": "replicated-route",
      "get": {
        "type": "RandomRoute",
        "children": [
          "PoolRoute|memcached-event",
          "PoolRoute|memcached-notification"
        ]
      }
    }
  }
}

jmswen avatar Feb 07 '18 04:02 jmswen

Hi Jon. Not quite. So here is what i want.

Ideally, we want two memcached pools, memcached-event with two servers containing the same data, and memcached-notification with two servers containing the same data. So two different pools each for a different function and team, and memcached servers in each pool must be synced.

So regarding reads and writes, this is what we want:

  • send gets to a random box from a pool (e.g., memcached-event with IP 10.1.9.100:11211). If request fails, get the data from any other box (still within the same pool, so now 10.1.9.101:11211 http://10.1.9.100:11211/).
  • send addd, sets and deletes to all hosts inside a pool (e.g., servers in the memcached-event)..

On Wed, Feb 7, 2018 at 12:46 PM, Jon Swenson [email protected] wrote:

It sounds like you want split read requests between the two pools (e.g., randomly pick one of the two pools on each request), but have writes/deletes replicate to both pools. The following config achieves that:

{ "pools": { "memcached-event": { "servers": [ "10.1.9.100:11211", "10.1.9.101:11211" ] }, "memcached-notification": { "servers": [ "10.1.9.200:11211", "10.1.9.201:11211" ] } }, "route": { "type": "OperationSelectorRoute", "operation_policies": { "add": { "type": "AllSyncRoute", "children": [ "PoolRoute|memcached-event", "PoolRoute|memcached-notification" ], "name": "replicated-route" }, "delete": "replicated-route", "set": "replicated-route", "get": { "type": "RandomRoute", "children": [ "PoolRoute|memcached-event", "PoolRoute|memcached-notification" ] } } } }

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/facebook/mcrouter/issues/229#issuecomment-363655948, or mute the thread https://github.com/notifications/unsubscribe-auth/APLx-v2NDR85G13XqtVbutVCiVRBzanRks5tSSqQgaJpZM4R6eRc .

-- Purpose is the key to success. You can easily be successful in the wrong things.

chilumbugeorge avatar Feb 07 '18 04:02 chilumbugeorge

Hi George,

I see that you're getting: "CRITICAL: Couldn't initialize from standalone flavor file /etc/mcrouter.conf" This may still mean that your config file is in malformed format. For example, a missing comma at the end of the "set" line. Your daemon listens on port 11212, so to test mcrouter on that machine (once you get rid of the CRITICAL line), do: printf "set abc 0 0 4\r\n1234\r\n" | nc localhost 11212 printf "get abc\r\n" | nc localhost 11212

Ori

orishu avatar Feb 07 '18 23:02 orishu

The last "set" in the pool shot not have a comma i believe this being a JSON format.

Even though the "CRITICAL line" is still being displayed, i tried running the two commands you gave me and it seems like it works:

george.chilumbu@mcrouter-1:~$ printf "set abc 0 0 4\r\n1234\r\n" | nc localhost 11212 STORED george.chilumbu@mcrouter-1:~$ printf "get abc\r\n" | nc localhost 11212 VALUE abc 0 4 1234 END

However, how do i know which memcached pool this "set" commands goes to? How can i write a set/get command to a specific memcached pool e.g., memcached-event using routes? May you please provide me with a specific example command?

chilumbugeorge avatar Feb 08 '18 08:02 chilumbugeorge

Also, when i run the command echo stats servers | nc 127.0.0.1 11212, i get only two servers for memcached-notification (the last configured pool in my config) listed and no info on memcached servers in other pools as shown below:

george.chilumbu@mcrouter-1:~$ echo stats servers | nc 127.0.0.1 11212 STAT 10.1.9.200:11211:ascii:plain:notcompressed-1000 avg_latency_us:522.410 pending_reqs:0 inflight_reqs:0 avg_retrans_ratio:0 max_retrans_ratio:0 min_retrans_ratio:0 closed:1; found:1 notfound:1 stored:1 STAT 10.1.9.201:11211:ascii:plain:notcompressed-1000 avg_latency_us:620.000 pending_reqs:0 inflight_reqs:0 avg_retrans_ratio:0 max_retrans_ratio:0 min_retrans_ratio:0 closed:1; stored:1 END

And when i run echo "stats all" | nc 127.0.0.1 11212, i only get number of servers listed as 2:

george.chilumbu@db-mcrouter-1:~$ echo "stats all" | nc 127.0.0.1 11212 STAT version 36.0.0-master mcrouter STAT commandargs --debug-fifo-root /var/lib/mcrouter/fifos --stats-root /var/lib/mcrouter/stats -L /var/log/mcrouter/mcrouter.log -p -f -f /etc/mcrouter.conf -L /var/log/mcrouter/mcrouter.log -p 11212 --send-invalid-route-to-default --fil e-observer-poll-period-ms=1000 --file-observer-sleep-before-update-ms=100 /etc/mcrouter.conf STAT pid 15494 STAT parent_pid 1 STAT time 1518078980 STAT uptime 2656 STAT num_servers 2

Seems like its only checking the last pool and and ignoring the first pool. Whats the deal with, any suggestions?

chilumbugeorge avatar Feb 08 '18 08:02 chilumbugeorge

@chilumbugeorge, you cannot list the same operation more than once in operation_policies, as that creates ambiguity as to how that operation should be routed. You will need to use PrefixSelectorRoute and prefix your keys so that mcrouter can distinguish whether requests should go to the memcached-event or memcached-notification pool. E.g.,

"route": {
  "type": "PrefixSelectorRoute",
  "policies": {
    "event_": {
      // Route for memcached-event keys.
      // Keys prefixed with "event_" go down this route.
    },
    "notification_": {
      // Route for memcached-notification keys
    }
  }
  "wildcard": "ErrorRoute|Got key with unexpected prefix"
}

jmswen avatar Feb 08 '18 15:02 jmswen

@chilumbugeorge - you should use a JSON validator to make sure that your config file is well formed. Try this one: https://jsonlint.com/

orishu avatar Feb 08 '18 16:02 orishu

Thanks Ori for the tool. Very useful indeed.

On Fri, Feb 9, 2018 at 12:20 AM, Ori Shalev [email protected] wrote:

@chilumbugeorge https://github.com/chilumbugeorge - you should use a JSON validator to make sure that your config file is well formed. Try this one: https://jsonlint.com/

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/facebook/mcrouter/issues/229#issuecomment-364164354, or mute the thread https://github.com/notifications/unsubscribe-auth/APLx-vSYcGFyM_VSx-lVBQ3IF7_I806Nks5tSx7KgaJpZM4R6eRc .

-- Purpose is the key to success. You can easily be successful in the wrong things.

chilumbugeorge avatar Feb 08 '18 16:02 chilumbugeorge

So can i do either:

"route": { "type": "PrefixSelectorRoute", "policies": { "event_": "PoolRoute|memcached_event", "notification_": "PoolRoute|memcached_notification" }, "wildcard": "ErrorRoute|Got key with unexpected prefix" }

So requests with key prefix "event_" will be sent to pool 'memcached_event', and requests with key prefix "notification_" will be sent to pool 'memcached_notification'

OR

"route": { "type": "OperationSelectorRoute", "operation_policies": { "event_": { "add": "AllSyncRoute|Pool|memcached-event", "delete": "AllSyncRoute|Pool|memcached-event", "get": "LatestRoute|Pool|memcached-event", "set": "AllSyncRoute|Pool|memcached-event" }, "notification_": { "add": "AllSyncRoute|Pool|memcached-notification", "delete": "AllSyncRoute|Pool|memcached-notification", "get": "LatestRoute|Pool|memcached-notification", "set": "AllSyncRoute|Pool|memcached-notification" }, "wildcard": "ErrorRoute|Got key with unexpected prefix" } }

Ideally, add, delete, get and set requests prefixed with "event_" keys will be sent to pool 'memcached_event', and add, delete, get and set requests with key prefix "notification_" will be sent to pool 'memcached_notification'

???

Or Using your code example specifically, would the config be more like this?:

"route": { "type": "PrefixSelectorRoute", "policies": { "event_": { "route": "PoolRoute|memcached_event" }, "notification_": { "route": "PoolRoute|memcached_notification" } } "wildcard": "ErrorRoute|Got key with unexpected prefix" }

On Thu, Feb 8, 2018 at 11:34 PM, Jon Swenson [email protected] wrote:

@chilumbugeorge https://github.com/chilumbugeorge, you cannot list the same operation more than once in operation_policies, as that creates ambiguity as to how that operation should be routed. You will need to use PrefixSelectorRoute and prefix your keys so that mcrouter can distinguish whether requests should go to the memcached-event or memcached-notification pool. E.g.,

"route": { "type": "PrefixSelectorRoute", "policies": { "event_": { // Route for memcached-event keys. // Keys prefixed with "event_" go down this route. }, "notification_": { // Route for memcached-notification keys } } "wildcard": "ErrorRoute|Got key with unexpected prefix" }

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/facebook/mcrouter/issues/229#issuecomment-364149014, or mute the thread https://github.com/notifications/unsubscribe-auth/APLx-tSNn4wpFw18lH2jun1poxa2rEXsks5tSxQGgaJpZM4R6eRc .

-- Purpose is the key to success. You can easily be successful in the wrong things.

chilumbugeorge avatar Feb 08 '18 17:02 chilumbugeorge