Code Reading - Learn ❤️ RabbitMQ - Part 1

Published Thursday, August 31, 2017

Notes from Flatiron School engineering team’s code reading by @joshrowley, part 1.

Concepts

Fuzzy overview of stuff we’ve heard about RabbitMQ

  • Written in Erlang
  • Queues, Exchanges, Channels, Subscriptions
  • Publishers, Consumers
  • Connections
  • Fanouts
  • Self-healing
  • Distributed

Actual overview

  • Message broker originally built by JPMorgan Chase. Built 2004, released 2007.
  • Designed as a way to solve problems with integration points between many different bank systems. Needed standardized way of sending messages between apps.
  • Goals: reliability, guaranteed message delivery between apps, uptime
  • Solution: AMQP (protocol)

What is AMQP?

  • Runs on Erlang OTP
  • Queues where you persist messages
  • Consumers who consume messages
  • Acks inform queue when message is processed
  • Whole thing ensures streams of data are processed as intended
  • Any client can use AMQP, so why roll your own?
  • Decouples publishers from consumers
Producer - app that publishes messages (Ironbroker, Ironboard, Flatchat)

|
|  -  TCP Connection (ongoing 2-way connection)
V

Broker (RabbitMQ)  
  - Queues ("mailboxes"): hold messages, wait for consumer to connect and take message off queue, load balances
  - Exchanges: publishers publish message to exchange, exchange routes to queue(s)
  - Bindings: rules you define for how exchanges publish messages to queues
  - Connections: can be multi-plexed into multiple channels

|
|  -  TCP Connection (ongoing 2-way connection)
V

Consumer - app that receives messages (Ironworker, Sneakers)

Our Publishers

  • Ironbroker
  • Ironboard
  • Flatchat

Our Consumers

  • Ironworker
  • Sneakers

Direct vs Fanout vs Topic Exchanges

  • Direct exchange: routes based on routing key
  • Fanout exchange: routing keys DON’T MATTER, publishes out to all queues (broadcast)
  • Topic exchange: routes based on routing key, queues bind to exchange on glob operators
# Fanout Exchange

Publisher
   |
   |
   V
Exchange
|  |  |
|  |  |
V  V  V
Q  Q  Q  (messages broadcast to all bound queues)

# Topic Exchange
# see ActivityFeedPublisher

        Publisher
           |
           |
           V
        Exchange
|          |          |
|          |          |
V          V          V
Q          Q          Q
usa.e*  usa.west   usa.north*

Code

Sneakers (aka our “workers”) uses a gem called Bunny gem.

# connect to rabbit with default settings
conn = Bunny.new
conn.start

# create channel on the connection
# channels are a way for a TCP connection (expensive) to be shared in a lightweight way
channel = conn.create_channel

# create queue
# consumers get messages from queues
queue = channel.queue('hello_world', auto_delete: true)

# subscribe to queue
# the rabbitmq server will push messages down the channel when the queue receives messages
queue.subscribe do |delivery_info, metadata, payload|
  puts "queue #{queue.name} received message payload: #{payload}"
end

# create exchange using the default exchange
exchange = channel.default_exchange

# when not using default exhange, would need to bind queue to exchange
# queue = channel.queue('hello_world')
# queue.bind(non_default_exchange, routing_key: 'cool key')

# send some messages
while true
  payload = "Hello World!"

  # by default, AMQP uses queue names for routing keys
  routing_key = queue.name
  exchange.publish(payload, routing_key: routing_key)
  puts "publishing to the default exchnage with routing key #{routing_key} and message payload: #{payload}"
  sleep 1
end

Day to Day Usage

  • Sneakers
  • Ironbroker - translate HTTP to AMQP (publisher)

More next week!

Clustering

Next week!

vs Kafka

Next week!

Resources

  1. RabbitMQ
  2. Bunny gem
  3. Sneakers