RPC - Part: Discovery Services
What is a Discovery Service? In SOA architecture (distributed systems), services need to find each other. For example, some web service might need to find a caching service, etc.
A Service Discovery system provides a mechanism for:
- Services to register their availability
- Locating a single instance of a particular service
- Notifying when the instances of a service change
It can look like DNS can be used for this - but it is nowhere near flexible enough for services that are constantly changing. It it far too slow in synchronizing, and doesn’t provide enough information about service, and much more - it was simply not meant to be used for that, really.
Apache Zookeeper
braitz/node-zookeeper-thrift · GitHub
Distributed Coordination with Zookeeper Chubby has in effect replaced DNS within the Google infrastructure!
Zookeeper are both much more than a distributed lock service. In fact, it may be better to think of them as implementations of highly available, distributed “metadata” filesystems.
ZooKeeper is a “coordination service” for distributed applications, originally developed by Yahoo. Name service and configuration are two of the primary applications of ZooKeeper.
ZooKeeper Recipes and Solutions Apache ZooKeeper
Norbert is a good example from a scalable production system. I general, it integrates Netty, Protocol Buffers and Zookeeper into a lightweight framework for running clustered services. Protocol Buffers are used to specify your service API, Netty implements transport-layer abstractions and Zookeeper is essentially a fault-tolerant discovery service.
Zookeper with Thrift
As you start to scale an infrastructure with Thrift services, you may find that putting all of your Thrift server IP/port combinations in a configuration file that your clients read is just not best.
If you have an environment where Thrift servers go down for maintenance (or crash), or you add more capacity on the fly, a dynamic way to manage the state of all of the services is needed.
Enter Apache ZooKeeper, a distributed, fault tolerant, highly available system for managing configuration information, naming, and more (distributed synchronization, anyone?).
Zookeeper guaranties that a write either fails or is consistent and that a read will get you the most recent state
Further reading Zookeeper for web developers
Libraries
You can even connect to ZookKeeper with REST. This REST ZooKeeper gateway is useful because most of the languages have built-in support for working with HTTP based protocols.
[zookeeper/src/contrib/rest](https://github.com/apache/zookeeper/tree/trunk/src/contrib/rest)
A PHP extension for interfacing with Apache ZooKeeper A Node.js client for Apache Zookeeper
Web Admin UI / Dashboard for ZK A zookeeper web administrator in node.js
Curator is a set of Java libraries that make using Apache ZooKeeper much easier
Service Discovery · Netflix/curator Wiki
Doozer, Zookeeper alternative
Written in Go! A consistent distributed data store for small amounts of extremely important data. Doozer
If ZooKeeper isn’t your thing, you should definitely check out Doozerd. Take a good look at it, it’s a nice, single binary distributed coordination service developed by Heroku.
Doozer client for node.js dannycoates/doozerjs
Service Discovery
mDNS / Avahi / Bonjour / Zeroconf as service discovery
Zeroconf networking based on ZeroMQ gaspard/zconf · GitHub Use service names instead of fixed IP and port with ZeroMQ
Let’s speculate that I’m developing a RPC over TCP application and I need to invoke a named service, say “helloworld”. I have a service discovery subsystem capable of resolving the “helloworld” service and telling my client which nodes provide this service and on which port to connect, etc. It’s possible that my service discovery subsystem is stale and responds with a node that has been taken down. If my RPC client detects a faulty node, it should failover to other nodes until it finds one that is reachable.
0MQ is definitely meant for end-to-end communication. If there are middle nodes, they are accidental and should be transparent to the end user. Think of TCP. It appears as end-to-end service, however, the packets can be transported via multiple network nodes.
The pro of brokerless model is performance. Lower latency and higher bandwidth available. Also, there’s no single bottleneck.
The pro of box in the middle is that there’s a single address to connect to.
Re: [zeromq-dev] RPC design pattern
Zookeeper for discovery & coordination of services using ZeroMQ. ZeroMQ works very well in combination with Apache Zookeeper [1] for service disc… | Hacker News
http://static.usenix.org/events/usenix10/tech/full_papers/Hunt.pdf
mDNS (Multicast DNS) and DNS-SD
Only inside a LAN. We could sometimes need an Multi LAN communication.
Host Discovery Using mDNS allows a client to determine the IP address of a given host (a host name) without the direct help of a centralised DNS server.
Service discovery DNS-SD allows to build a list of hosts providing a given service.
ZeroMQ internal discovery patterns
Of all the different patterns, the two that stand out are the Majordomo pattern (for broker-based reliability), and the Freelance pattern (for brokerless reliability).
ØMQ - The Guide - ØMQ - The Guide Brokerless Reliability (Freelance Pattern)
Others
high availability - HA and centralized config storage for zeromq - alternatives to zookeeper and doozer - Stack Overflow ZooKeeper alternatives? (cluster coordination service) - Stack Overflow
Accord is a coordination service (like ZooKeeper) written in C. It’s built for high write loads. collie/accord · GitHub
Noah is an application registry inspired by Apache ZooKeeper. lusis/Noah
[zeromq-dev] Paxos & ZeroMQ You could consider zpax, it’s similar to zookeeper/doozer in basic concept but it’s intended for direct embedding into the target application.
The problem here is - there is no PHP solution.
I could write my file-based discovery (not clustered), similar to this solution in node.js: nodester proxy
Load balancing with Zookeeper
FB303 How are people load balancing Thrift based services?
java - Load balanced service using Zookeeper and Thrift - Stack Overflow
Norbert by LinkedIn is something similar, but uses Protobuf as transport, an everything is implemented in Scala. Norbert - The LinkedIn Data Team
commons/src/java/com/twitter/common/thrift/ThriftFactory.java at master · twitter/commons · GitHub
Interesting projects and idead with Zookeeper
-
Monitor a particular service on a host machine and register that machine as a provider of that service at a given path on the ZooKeeper service Nextdoor/zkwatcher
-
There is a Puppet module for installing zkWatcher. Nextdoor/puppet_zkwatcher But this module and plugin serves one more function: a method for building Puppet manifests with dynamically-obtained ZooKeeper server lists. Great!
Zookeeper:
fluent-plugin-scribe/lib/fluent/plugin/thrift at master · fluent/fluent-plugin-scribe · GitHub
Distributed Coordination with Zookeeper - igvita.com - http://www.igvita.com/2010/04/30/distributed-coordination-with-zookeeper/ ZooKeeper Recipes and Solutions - http://zookeeper.apache.org/doc/trunk/recipes.html#sc_outOfTheBox ZooKeeper Administrator’s Guide - http://zookeeper.apache.org/doc/r3.3.3/zookeeperAdmin.html#sc_zkMulitServerSetup Vagrant - Documentation - Bridged Networking - http://vagrantup.com/v1/docs/bridged_networking.html How To: Make VirtualBox Use Your Router’s DHCP to get an IP Address in Linux | r3dux.org - http://r3dux.org/2009/09/how-to-make-virtualbox-use-your-routers-dhcp-to-get-an-ip-address-in-linux/ How do I set permanent IP for my VirtualBox using Bridge adapter? - Super User - http://superuser.com/questions/384719/how-do-i-set-permanent-ip-for-my-virtualbox-using-bridge-adapter How to run Virtualbox on bridged and host-only networks? - Ask Ubuntu - http://askubuntu.com/questions/113604/how-to-run-virtualbox-on-bridged-and-host-only-networks yuxel/thrift-examples - https://github.com/yuxel/thrift-examples/ thrift/test/php/TestNonblockingServer.php at trunk · volca/thrift - https://github.com/volca/thrift/blob/trunk/test/php/TestNonblockingServer.php Thrift PHP Server - Lexicon - Coder Forum - http://www.easy-coding.de/wiki/php/thrift-php-server.html#headline5 tenorviol/php-thrift-client - https://github.com/tenorviol/php-thrift-client thrift php server - Google Search - https://www.google.com/search?q=php+server+socket+example&oq=php+server+socket+example&sugexp=chrome,mod=11&sourceid=chrome&ie=UTF-8#q=thrift+php+server&hl=en&prmd=imvns&ei=1q9tUNrYCMfOsgb50IHoBA&start=10&sa=N&fp=1&biw=1366&bih=643&cad=b&sei=33huUKu1LYjGswbRz4CQDw&sei=cO5uUJ25O4betAa1sIHoDA&sei=--RyUPqvK4LctAbliYHQDA&bav=on.2,or.r_gc.r_pw.r_cp.r_qf.&sei=4yFzUKqNLIbBtAayw4HgAg thrift/tutorial/php/runserver.py at trunk · apache/thrift - https://github.com/apache/thrift/blob/trunk/tutorial/php/runserver.py php实现的thrift socket server – 某人的栖息地 - http://www.ooso.net/archives/537 Thrift Developer Mailing List - http://comments.gmane.org/gmane.comp.lib.thrift.devel/6348 [Thrift-user] Multiple interfaces handled by the same TServer - Grokbase - http://grokbase.com/t/thrift/user/112en4qqsx/multiple-interfaces-handled-by-the-same-tserver Thrift and ZooKeeper « chris moos’s blog - http://chrismoos.com/2011/05/25/thrift-and-zookeeper/ ZooKeeper: Because Coordinating Distributed Systems is a Zoo - http://zookeeper.apache.org/doc/r3.4.4/ ZooKeeper Administrator’s Guide - http://zookeeper.apache.org/doc/r3.4.4/zookeeperAdmin.html#sc_zkMulitServerSetup install apache zookeeper ubuntu - Google Search - https://www.google.com/search?q=install+apache+zookeeper+ubuntu&oq=install+apache+zookeeper+ubuntu&sugexp=chrome,mod=11&sourceid=chrome&ie=UTF-8 ZooKeeper Administrator’s Guide - http://archive.cloudera.com/cdh4/cdh/4/zookeeper/zookeeperAdmin.html#sc_zkMulitServerSetup Test web site concurrent connections: tools ? - Spiceworks - http://community.spiceworks.com/topic/231008-test-web-site-concurrent-connections-tools