Programmatically creating shards.yml?

Question

I'm using Octopus to separate the reads and writes of my Rails application. I have the configuration for the whole application in a JSON file and a wrapper class for ease of access.

Now I'd like to configure octopus' slave connections by generating the shards.yml file, so that I can set the number of configured slaves in my JSON config file.

This works:

<%
# See config.json file for MyApp-specific configuration of web services
require 'myapp/config.rb' 
%>

octopus:
  environments:
    - <%= MyApp::Config::deployment %>
  replicated: true
  fully_replicated: true 
  development:
    slave1:
      host: <%= MyApp::Config::DatabaseSlaves.host(0) %>
      adapter: <%= MyApp::Config::DatabaseSlaves.adapter(0) %>
      database: <%= MyApp::Config::DatabaseSlaves.database(0) %>
      username: <%= MyApp::Config::DatabaseSlaves.username(0) %>
      password: <%= MyApp::Config::DatabaseSlaves.password(0) %>
      reconnect: <%= MyApp::Config::DatabaseSlaves.reconnect(0) %>

And this doesn't:

<%
# See config.json file for MyApp-specific configuration of web services
require 'myapp/config.rb' 
%>

octopus:
  environments:
    - <%= MyApp::Config::deployment %>
  replicated: true
  fully_replicated: true 
  <%= MyApp::Config::deployment %>:
  <% counter = 1
    while counter <= MyApp::Config::DatabaseSlaves.count do %>
    slave<%= counter.to_s %>:
      host: <%= MyApp::Config::DatabaseSlaves.host(counter - 1) %>
      adapter: <%= MyApp::Config::DatabaseSlaves.adapter(counter - 1) %>
      database: <%= MyApp::Config::DatabaseSlaves.database(counter - 1) %>
      username: <%= MyApp::Config::DatabaseSlaves.username(counter - 1) %>
      password: <%= MyApp::Config::DatabaseSlaves.password(counter - 1) %>
      reconnect: <%= MyApp::Config::DatabaseSlaves.reconnect(counter - 1) %>
  <%
      counter++
    end
  %>

Giving me an error when I run Webrick or Rake tasks on the database:

SyntaxError ((erb):23: syntax error, unexpected keyword_end
(erb):25: syntax error, unexpected $end, expecting keyword_end
; _erbout.force_encoding(__ENCODING__)
                                  ^):
/usr/lib/ruby/1.9.1/erb.rb:838:in `eval'
/usr/lib/ruby/1.9.1/erb.rb:838:in `result'
/home/jriepshoff/.bundler/ruby/1.9.1/octopus-4435a53054ee/lib/octopus.rb:22:in `config'
/home/jriepshoff/.bundler/ruby/1.9.1/octopus-4435a53054ee/lib/octopus.rb:64:in `environments'
/home/jriepshoff/.bundler/ruby/1.9.1/octopus-4435a53054ee/lib/octopus.rb:38:in `enabled?'
/home/jriepshoff/.bundler/ruby/1.9.1/octopus-4435a53054ee/lib/octopus/model.rb:51:in `should_use_normal_connection?'
/home/jriepshoff/.bundler/ruby/1.9.1/octopus-4435a53054ee/lib/octopus/model.rb:59:in `connection_with_octopus'
activerecord (4.0.1) lib/active_record/migration.rb:792:in `current_version'
activerecord (4.0.1) lib/active_record/migration.rb:800:in `needs_migration?'
activerecord (4.0.1) lib/active_record/migration.rb:379:in `check_pending!'
activerecord (4.0.1) lib/active_record/migration.rb:366:in `call'
actionpack (4.0.1) lib/action_dispatch/middleware/callbacks.rb:29:in `block in call'
activesupport (4.0.1) lib/active_support/callbacks.rb:373:in `_run__2591721780440848137__call__callbacks'
activesupport (4.0.1) lib/active_support/callbacks.rb:80:in `run_callbacks'
actionpack (4.0.1) lib/action_dispatch/middleware/callbacks.rb:27:in `call'
actionpack (4.0.1) lib/action_dispatch/middleware/reloader.rb:64:in `call'
actionpack (4.0.1) lib/action_dispatch/middleware/remote_ip.rb:76:in `call'
actionpack (4.0.1) lib/action_dispatch/middleware/debug_exceptions.rb:17:in `call'
actionpack (4.0.1) lib/action_dispatch/middleware/show_exceptions.rb:30:in `call'
railties (4.0.1) lib/rails/rack/logger.rb:38:in `call_app'
railties (4.0.1) lib/rails/rack/logger.rb:20:in `block in call'
activesupport (4.0.1) lib/active_support/tagged_logging.rb:67:in `block in tagged'
activesupport (4.0.1) lib/active_support/tagged_logging.rb:25:in `tagged'
activesupport (4.0.1) lib/active_support/tagged_logging.rb:67:in `tagged'
railties (4.0.1) lib/rails/rack/logger.rb:20:in `call'
actionpack (4.0.1) lib/action_dispatch/middleware/request_id.rb:21:in `call'
rack (1.5.2) lib/rack/methodoverride.rb:21:in `call'
rack (1.5.2) lib/rack/runtime.rb:17:in `call'
activesupport (4.0.1) lib/active_support/cache/strategy/local_cache.rb:83:in `call'
rack (1.5.2) lib/rack/lock.rb:17:in `call'
actionpack (4.0.1) lib/action_dispatch/middleware/static.rb:64:in `call'
rack (1.5.2) lib/rack/sendfile.rb:112:in `call'
railties (4.0.1) lib/rails/engine.rb:511:in `call'
railties (4.0.1) lib/rails/application.rb:97:in `call'
rack (1.5.2) lib/rack/lock.rb:17:in `call'
rack (1.5.2) lib/rack/content_length.rb:14:in `call'
rack (1.5.2) lib/rack/handler/webrick.rb:60:in `service'
/usr/lib/ruby/1.9.1/webrick/httpserver.rb:138:in `service'
/usr/lib/ruby/1.9.1/webrick/httpserver.rb:94:in `run'
/usr/lib/ruby/1.9.1/webrick/server.rb:191:in `block in start_thread'

I'd be really glad if anyone could give some hints on how to generate the YAML. I'm not familiar with the syntax and did quite a lot of research including tests on my own but without any success.

Answer 1

Don't use ERB, instead use straight-Ruby. When we have complex YAML files to create at work, I start in Ruby, and have it emit the boilerplate:

require 'yaml'

# octopus:
#   environments:
#     - <%= MyApp::Config::deployment %>
#   replicated: true
#   fully_replicated: true 
#   development:
#     slave1:
#       host: <%= MyApp::Config::DatabaseSlaves.host(0) %>
#       adapter: <%= MyApp::Config::DatabaseSlaves.adapter(0) %>
#       database: <%= MyApp::Config::DatabaseSlaves.database(0) %>
#       username: <%= MyApp::Config::DatabaseSlaves.username(0) %>
#       password: <%= MyApp::Config::DatabaseSlaves.password(0) %>
#       reconnect: <%= MyApp::Config::DatabaseSlaves.reconnect(0) %>

config = {
  'octopus' => {
    'environments' => 'MyApp::Config.deployment',
    'replicated' => true,
    'fully_replicated' => true,
    'development' => {
      'slave1' => {
        'host'      => 'MyApp::Config::DatabaseSlaves.host(0)',
        'adapter'   => 'MyApp::Config::DatabaseSlaves.adapter(0)',
        'database'  => 'MyApp::Config::DatabaseSlaves.database(0)',
        'username'  => 'MyApp::Config::DatabaseSlaves.username(0)',
        'password'  => 'MyApp::Config::DatabaseSlaves.password(0)',
        'reconnect' => 'MyApp::Config::DatabaseSlaves.reconnect(0)',
      }
    }
  }
}

puts config.to_yaml

Which generates:

---
octopus:
  environments: MyApp::Config.deployment
  replicated: true
  fully_replicated: true
  development:
    slave1:
      host: MyApp::Config::DatabaseSlaves.host(0)
      adapter: MyApp::Config::DatabaseSlaves.adapter(0)
      database: MyApp::Config::DatabaseSlaves.database(0)
      username: MyApp::Config::DatabaseSlaves.username(0)
      password: MyApp::Config::DatabaseSlaves.password(0)
      reconnect: MyApp::Config::DatabaseSlaves.reconnect(0)

Once I have the basic template I can either modify it by hand or flesh out the code to actually generate everything I need. I do this because some applications have a lot of configuration parameters, and having to generate that configuration by hand would be too error-prone. Instead a YAML-file generator can do it in a second then we can tweak as necessary.