Gmail Security Blocking Rails Emails

So as many people, I’ve setup my Rails app to send emails for various things. My particular use case was sending an email when a new user signed up. I used a free gmail account as the email address to send from.  However, I kept running into a problem with the gmail security settings blocking the email because it looked like a suspicious user, because the heroku server happened to be in Virginia, where I don’t live.

So, how to fix this? I searched and found almost nowhere where anyone had described how to fix this. Maybe I’m the only one who had this problem, but it seems like a very common thing.

First step to make sure that your app doesn’t crash if you have this type of email error is to set config.action_mailer.raise_delivery_errors = false in your config/environments/production.rb file. It can be helpful to set this to true in your development or testing environments, but in production you want these to fail silently so they don’t break your app.

Ok, now the trick to fix the security settings in Google. Counterintuitively, you need to turn ON 2-factor authentication. Go to and login with the account you want to send email from in your Rails app.  Scroll to “Signing in to Google”, and turn on 2-Step verification. You need that turned on so that you can then setup “App passwords”. Next click on App passwords and you get this screen.

Screen Shot 2017-08-16 at 7.19.15 AM

Click on Select App, then Custom, and give it a useful name, and click “Generate”. Now you get this popup with your new app password.

Screen Shot 2017-08-16 at 7.19.44 AM

Screen Shot 2017-08-16 at 7.20.14 AM

When this password is used, it will not block it based on location, etc.

Now add this password to your Rails app as the password it uses to login to gmail, rather than your usual password. I use the figaro gem for environment variables, so I updated my GMAIL_PASSWORD environment variable in my application.yml file.

That’s it! Hopefully it was useful!

helpful links:

Rails Paperclip Attachments Amazon Cloudfront CDN with Signed URLs using the Cloudfront-signer Gem and Tricky YAML

So here’s what I was trying to do. I have a Rails app where users upload attachments. In my case they are pdf’s, but I think this still applies if they were images or any other file type. I use the paperclip gem to handle the pdf attachments, and store them on Amazon S3 bucket cloud storage. This was all working fine. However, I wanted to make 2 improvements:

  1. Use a CDN such as Amazon Cloudfront to deliver these attachments faster globally. See this blog post that first put me on to the idea.
  2. I originally made the S3 bucket public to serve these assets, but I wasn’t happy with this. I wanted more security and privacy for these attached files. So after setting up the Amazon Cloudfront CDN, I wanted to use signed and expiring URL’s to protect the content.

While initially this seemed like a somewhat common and straightforward thing to do, I ran into a lot of annoying little hiccups that took way too much time to figure out. I never found somewhere where the steps to do this were all in one place. So I’m writing this to hopefully do that.

First you need to pick your CDN. There are many options, including one that looks promising called Cloudinary (and their Attachinary library), which I might switch to later and do another post or update this one. For now I chose Amazon Cloudfront since it is well documented and has lots of stackoverflow support :).

So to create an Amazon CDN distribution, login to your AWS console and click “Create Distribution”, then select a “web” distribution. For “Origin Domain Name”, pick the S3 bucket where your files are stored. This is the ‘origin’ of the files for Cloudfront. I left the “origin path” blank, as its optional. Next, give it whatever ‘origin-ID’ you want to make it descriptive to yourself. In my case I wanted to select “Restrict Bucket Access”, which then disables the Amazon S3 URLs, so your content can only be viewed from the Cloudfront URL. Then “use an existing identity” for Origin Access Identify if you have one, or create a new one and give it a useful name. Then for simplicity and ease of use I chose “Yes, Update Bucket Policy” under “Grant Read Access”. This allows the Cloudfront distribution to read the files in your S3 bucket so it can distribute them. If you don’t have it update the policy automatically, you will need to manually update the bucket policy on your S3 bucket to something like this:

"Version": "2008-10-17",
"Id": "PolicyForCloudFrontPrivateContent",
"Statement": [
"Sid": "1",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::cloudfront:user/CloudFront Origin Access Identity xxxx"
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::bucketname/*"

I left the “Origin Custom Headers” blank. Setup the “Default Cache Behavior Settings” however you see fit. I chose to redirect HTTP to HTTPS – its best to not use HTTP whenever possible. Select “Yes” for “Restrict Viewer Access” to use signed URLs, and pick the appropriate signers. Everything else I left as the default options. For “Distribution Settings” you can use CNAMEs to set up URLs to use your custom domain, but I didn’t do this. I did choose to use the default Cloudfront SSL Certificate, again for simplicity. Everything else I left as defaults, and you’re all set up! It takes a few minutes for your distribution to be fully enabled after you create it.

Awesome, we now have a Cloudfront distribution setup to deliver the assets from our S3 bucket, making it much faster. And we can use signed URLs to protect the privacy of these files. Let’s do that next.

The basic structure of signed URL’s is as follows:


The example in the AWS documentation is: vDh6hQ73lDx~-ar3UocvvRQVw6EkC~GdpGQyyOSKQim-TxAnW7d8F5Kkai9HVx0FIu- 5jcQb0UEmatEXAMPLE3ReXySpLSMj0yCd3ZAB4UcBCAqEijkytL6f3fVYNGQI6&Key-Pair-Id=APKA9ONS7QCOWEXAMPLE

Ok, so the URL, query parameters, and expiration are easy. But we need to get the Signature and Key-Pair-ID.

Let’s do the Key-Pair-ID first. In your AWS console, go to the Security Credentials section, and open Cloudfront Key Pairs. Create a new key pair, then download the private and public keys. Of course, save the private key in a secure location and don’t share it with anyone. We’ll come back to them in a minute.

Now we need to add Trusted Signers, to give someone the authority to create signed URL’s. For web distributions, you need to create a Trusted Signer based on each cache behavior you have setup. For me, there was just one. Go to your AWS Cloudfront console and select the distribution settings for the distribution you’re working with. Go the Behaviors tab and select the behavior in the list – for me the only was was the redirect HTTP to HTTPS behavior. Select Edit, and choose Restrict Viewer Access, and select yourself or whoever you wish as your Trusted Signer. This account can create signed URLs.

Ok, now that we have Trusted Signers assigned, we can go ahead and create our signed URLs. The Amazon documentation for how to create signed URLs is pretty good, and you can do it manually yourself. I found this gist that pretty much gives you the ruby code to create your signed URLs – its only a few lines. But wait, this is Rails, there’s gotta be a gem for that, right? Yep, there is! It’s called cloudfront-signer.

Install the gem and run the initializer:

bundle exec rails generate cloudfront:install

Now open the created config/initializers/cloudfront_signer.rb file. This is where we will put our credentials for creating the signed URL.

As described above, we need the Cloudfront Key-Pair-ID to append to the end of the URL. This was a bit confusing for me, but your Key-Pair-Id is called “Access Key ID” in your Cloudfront Security Credentials page where you downloaded the private and public keys. See screenshot:

Screen Shot 2017-08-08 at 7.36.11 AM

Copy the text string of capital letters under “Access Key ID” (mine is blanked out in the screenshot for security) and paste it into the cloudfront-signer.rb file for “key_pair_id”:

Aws::CF::Signer.configure do |config|
config.key_path = '/path/to/keyfile.pem'
# or config.key = ENV.fetch('PRIVATE_KEY')
config.key_pair_id = 'XXYYZZ' # copied from your Cloudfront "Access Key ID"
config.default_expires = 3600

Ok, last thing we need for configuration is your private key that you downloaded in the steps above. You can either save the .pem file itself into your Rails project somewhere – making sure to add it to .gitignore so it doesn’t become public in your version control!! – or you can copy/paste the text of the key into an environment variable. I originally used the first option, but then when I wanted to push to heroku in production, I didn’t know how to reference this file in production since I didn’t have it in version control, so I decided to use an environment variable.

Sounds simple right? Well, usually, but here’s where I ran into more trouble. I didn’t know that actually the \n line breaks in your private key file are critical, and can’t be removed. (I learned that from this unrelated github issue) So to put a multi-line environment variable in an application.yml file (as is done with the figaro gem in Rails) there are some fancy yaml tricks that you need to do. If you don’t you keep getting some weird errors, such as Neither PUB key nor PRIV key: nested asn1 error (OpenSSL::PKey::RSAError) because OpenSSL tried to sign the key without the \n line breaks, making it an invalid key. It took me a long time to debug this error, but hopefully this helps save you time.

Based on what I learned from this StackOverflow question, you need to add |- in front of your key in the application.yml file to keep the line breaks, but not create one at the end of the file. So the code in your application.yml will look like this:

#long string key here

That sneaky little |- is critical!

Ok, so we have all our credentials set up. If you’re not using the gem, you need to do a bit more manually. Signed Cloudfront URL’s use a policy statement. You can create a custom one with various parameters, or use a canned policy. I just used a canned policy statement. The example in the AWS documentation is:

"Resource":"base URL or stream name",
"AWS:EpochTime":ending date and time in Unix time format and UTC

Our cloudfront-signer gem conveniently creates the policy statement for us.

To use the gem to create a signed URL, just call Aws::CF::Signer.sign_url {your_attachment_url}, expires: + 600. In my case I put this in my show.html page where I wanted to display the attached pdf in an iframe.

Now one more tricky note was how to add our multiline environment variable to heroku while preserving the line breaks. Based on this StackOverflow question, you can use the cat function in the terminal to save it as a variable in your terminal session, then use that to create the variable on heroku:

testvar=$(cat myfile.txt). Then heroku config:add EC2_PRIVATE_KEY="$testvar"

Whew, more complicated than it sounded at the start, huh? Hopefully this was useful and saved you some headaches!

Here’s a list of helpful links:

Add Amazon Cloudfront as CDN from S3 Storage

I have a small app that was serving file attachments directly from S3. Although I wasn’t doing any processing, it’s still preferred to deliver those over some type of CDN. Here I’ll lay out how I added an Amazon Cloudfront CDN so now my S3 assets are served from there instead of directly from S3.

I’m assuming you already have S3 setup to store your attachment files. This post will talk about the setup using the Paperclip gem, but you can use others like Shrine or Cloudinary.

So first you need to setup an Amazon Cloudfront distribution that will be linked to your S3 bucket.

Go to your Amazon AWS console and create a new Cloudfront distribution. There are loads of settings, and most of them I just left as the default. For the first one “Origin domain name”, click in the box and it will give you the list of all the S3 buckets in your account. Select the one you want. It will auto-fill the Origin-ID box. You can leave the Origin Path blank, or fill it in with a specific folder in your bucket that you want to use. For Protocol Policy, I switched it to redirect HTTP to HTTPS to ensure everything is using HTTPS. Everything else I left as the default. You can change them later. Your distribution is now created. It will take a few minutes to be deployed.

Note that your file needs to be set to public access.

To test it, go to your Amazon S3 bucket, and get the link to a file. It will look like Open it in the browser. Now go to your cloudfront distribution, and replace “” with your cloudfront url, “”. You should be able to load the file in the browser from the cloudfront url.

What was not obvious to me is that when using the cloudfront url, you don’t need the bucket name. So for the S3 URL of

the Cloudfront url is

No we can go to our Rails app and setup Paperclip to deliver the attachments from Cloudfront. In config/environments/production.rb

config.paperclip_defaults = {
:storage => :s3,
:url => ':s3_alias_url',
:s3_host_alias => "",
:path => '/:class/:attachment/:id_partition/:style/:filename',
:s3_credentials => {
:bucket => ENV['AWS_BUCKET'],
:access_key_id => ENV['AWS_ACCESS_KEY_ID'],
:secret_access_key => ENV['AWS_SECRET_ACCESS_KEY']
:s3_region => ENV['AWS_REGION'],
:endpoint => ENV['AWS_ENDPOINT'],
:s3_protocol => 'https',

With the key being the :s3_host_alias and :url and :path settings.

And you’re all set! You can also set up Cloudfront to deliver your actual Rails app assets (javascript and css), but I’ll do that in a later post.

credit here

Ghostery extension blocking Google Analytics

I fought for a long time trying to get Google Analytics set up. I tried using the Google Tag Assistant Chrome extension, which was very helpful. I wasn’t getting any errors, I was just not getting any hits, even when I would load the site myself. So I finally figured out what was happening by checking the console – my Ghostery extension was blocking the Google Analytics script from sending its data.

So if you’re having errors with your Analytics data, be sure to turn off any privacy Chrome extensions that may be blocking your scripts. Hope this helps save you some time!

Specify RVM Gemset for Each App

Using rvm is always a bit confusing for me – I never fully understand exactly how its working. One thing I learned recently was how to specifcy a specific gemset for an app. I think when you refresh a terminal session, rvm switches back to the default gemset. I was upgrading to Rails 5, and I wanted to specificy to use my rails5.0 gemset with my app for testing. Here’s how you do it:

 rvm --ruby-version use 1.9.3@my_app

where my_app is the name of your gemset. This creates two files in the root of your application: .ruby-version and .ruby-gemset. These files specify the ruby version and gemset to use for your app.

Hope that makes sense, and if its wrong please let me know!

credit here, here, and here

Push local non-master branch to Heroku

So I was in the process of upgrading my app to Rails 5 (link1, link2), and wanted to push it to my staging app on heroku. I created a new git local branch called Rails5 to do the upgrade locally. The I kept running git push staging3 master to try to push it to heroku, but when I ran heroku run rails --version --remote staging3 to check the version of rails on heroku, I could see it wasn’t updating. By default heroku only pushes the master branch. You have to run a special command to push a different local branch to the remote heroku master branch. Essentially heroku only has a master branch – so you need to push your non-master local branch to the heroku master branch. Apparently I didn’t know this key command on how to push a non-master branch to heroku:

git push heroku yourbranch:master

see heroku article here

Here’s two other helpful links (link1link2) for upgrading to rails 5

Rails Single Form with Multiple Instances of A Model

I wanted to have a single form in my Rails app that submitted and created multiple instances of a Model. Seems like this would be a relatively common thing to do, right? But I found it surprisingly difficult to find good examples that I could follow. I worked on this for a long time, and hopefully this will save you some time.

I originally tried the approach of simply having multiple identical forms on the same page, and using Javascript to recursively submit them (hidden to the user). I imagine this could work, but I had difficulty getting it working, and realized that this approach had problems: 1) It is very slow and inefficient and 2) its not scalable. And I felt “there has to be a better way”.

I tried to use the Surveyor gem but it is no longer in development, and I couldn’t get it working with the latest Ruby and Rails 5. Toward the end of my work I was told about the Reform gem, which looks like it could work, but I had already gotten things figured out so I haven’t tried that gem.

So, on to the specifics of the problem. I wanted to create an ‘exam’ for users, that would be on a single web page. They would answer several questions and get a score. The way I have this set up is with a Question, Response, and Answer models. The Answer model stores the correct answers, while the Response model stores a user’s responses. Now if anyone knows a better way to structure this, I’d love to know, but its what I’ve come up with so far.

class Question < ApplicationRecord
	# Associations
	has_many :answers

class Answer < ApplicationRecord
	# Associations
	belongs_to :question

class Response < ApplicationRecord
	# Associations
	belongs_to :user
	belongs_to :question

Ok, so on the exam page, I want to display all of the Questions, and provide inputs so the user can submit all of their Responses, one for each Question. I want to do this on one webpage, and have one form that submits all of the user’s Responses. There are multiple ways to approach this, and I found a very helpful list (this gist) from Ryan Bates (of Railscasts) and discussed in the Rails Google Group thread.

The way I chose to approach it, I wanted to get the params to look like this:

Parameters: {"utf8"=>"✓", "authenticity_token"=>"", 
    "0"=>{"response_value"=>"a", "user_id"=>"23", "question_number"=>"1"},
    "1"=>{"response_value"=>"b", "user_id"=>"23", "question_number"=>"2"},
    "2"=>{"response_value"=>"c", "user_id"=>"23", "question_number"=>"3"}
 "commit"=>"Submit Exam", "method"=>"post"}

This is similar to ‘approach 4’ in the Ryan Bates gist referenced above. Essentially, I wanted a single form to submit params where the multiple response objects are in a hash in the params, where they are ordered. They are referenced in the hash by a plural of the model, responses.

So for example, the first response is under {"responses"=>{"0"}}

{"response_value"=>"a", "user_id"=>"23", "question_number"=>"1"}

I didn’t know how to do this, but after a lot of reverse engineering, here’s what I learned. In order to make this work when the HTML form is submitted, we need the name value of each input to be correct and unique. The name attribute of an html input is what links it to how it is defined in the params. So we need our name attribute to be as such in the rendered HTML:

input id="responses_response_value_a" class="radio" 
name="responses[0][response_value]" type="radio" value="a"

So for each Question in the exam, the Response will correspond to a unique numbered value in the params hash. In the above example, it is the 0 in responses[0][response_value]. So for example, the HTML input for response number 3 is:

input id="responses_response_value_c" class="radio" 
name="responses[2][response_value]" type="radio" value="c"

Keep in mind that I’m numbering these starting from 0, as in indices of an array. But you could use any unique value. The reason for this is to separate them in the hash, so then in our controller we can iterate over the params hash, calling create on each response in the params (which we will get to below). I chose to make the unique identifiers simple integers starting from zero to make the iteration code in the controller easier, as you will see below.

Ok, so thats the HTML that we want to generate. Of course we could manually create that in our Rails view file in straight HTML. But that wouldn’t be very “Railsy”. So how do we create that in our erb view file in Rails? Well, we can use Rails methods…

But first, in our controller for the Exam page, we need to create an array of the multiple instances of the Responses model that we are going to create when we submit the form. We do this before we render the exam page. So in whichever controller handles the page where we have our exam, in my case its called pages_controller.rb, I want to create an empty Response for each Question in the exam. I do this be creating an array of empty response objects:

def exam_page
    @empty_responses = []

    Question.all.each do
        @empty_responses <<

Great, so now we have an array of empty Responses, one for each Question. Now in our view, we can iterate over that array, and create form inputs for each Response, corresponding to the values for all those empty responses, which will get saved when we submit the form. So in exam_page.html.erb

form_tag responses_path(method: :post) do
   @empty_responses.each do |response|
     fields_for 'responses[]', response, include_id: false do |r|
      # grab each question and display its text
      q = Question.where(question_number:                    
                @empty_responses.index(response) + 1).first
            q.answers.order("answer_text"), :answer_text, :answer_text, 
            include_hidden: false) do |r|                 
               r.radio_button(class: 'radio' + 
               @empty_responses.index(response).to_s, name:
               'responses[' + (q.question_number - 1).to_s +   
               '][response_value]') + r.label(:class => 'radio-label')
      r.hidden_field :user_id, value:,
             name: 'responses[' + @empty_responses.index(response).to_s
              + '][user_id]'
      r.hidden_field :question_number, value: q.question_number,
             name: 'responses[' + @empty_responses.index(response).to_s  
            + '][question_number]'
  submit_tag 'Submit Exam', class: 'btn btn-primary btn-xl'

(Note that I removed the erb tags because I had trouble getting WordPress to display them correctly)
Ok, a lot going on here, lets break it down. So we use the form_tag for a new Response. I use the form_tag instead of form_for, as form_forseemed to be more difficult to get to handle multiple Responses, as it typically only accepts one. form_tag seemed to be more vanilla HTML.

In the next line we iterate over our array of empty responses that we created in the controller. Next we use fields_for to tell Rails that we are creating inputs for something different than a simple direct ‘one level deep’ Response. Remember we are going ‘two levels deep’ in our params, to include multiple Responses in our params hash. As this is not the default behavior, using fields_for allows us to tell Rails that we are creating input values for something other than what the form is for in the first line with the form_tag. Of course fields_for is often used if you want to include input values for a different Model, but in our case we are co-opting it to help us submit multiple instances of the same Model. We use "responses[]" to get us to the ‘second level’ in the params hash. So this means that any input values that come after our fields_for "responses[]" will correspond in the params hash to {responses=>{}}.

And thats how our name value on the HTML input becomes "responses[X][response_value]". Make sense?

So that’s the basics. However, since in my case I’m creating multiple choice questions with radio buttons, it ads some extra complications (of course!). I use the Rails form helper collection_radio_buttons. Lets go through each of these parameters as this was quite confusing to me initially. :response_value is the field in the database of the Response model that we want this data to be saved to. q.answers.order("answer_text") is the list of Answers, one for each radio button. The first :answer_text is the value that we want the radio button to be associated with, and the second :answer_text is the text that we want to display next to each radio button. Theinclude_hidden: false is an option that we pass in, which I’ll discuss below. Whew!

Here again, the name value is critical for us to get the HTML code right so that the Response gets put correctly into the params when submitted to the server. So here we specify the name attribute for each radio button. We need the iterate through and create the indices to be used in our params hash, as discussed above. Thats why I use the code: name: 'responses[' + (q.question_number - 1).to_s + '][response_value]'. So this gives us the "responses[X][response_value]" that we wanted in the HTML. The -1 is because I numbered my questions starting at 1, but I wanted to number the responses in the hash starting at 0. Now its important to note that for radio_buttons, the group of radio_buttons that are grouped together as a single set of choices must have the same name attribute.

The id value in the HTML is also important, as you can see it also links us to 2 levels deep by saying responses_response_value_c where ‘c’ is the value that is going to be submitted by choosing that input. This is automagically created by the rails helpers, and I found I didn’t need to alter it.

Here’s what the ultimate HTML looks like for the radio button options for the Response to the first Question:

input class="radio0" name="responses[0][response_value]" type="radio" 
      value="a" id="responses_response_value_a"
label class="radio-label" for="responses_response_value_a">"a"
input class="radio0" name="responses[0][response_value]" type="radio" 
      value="b" id="responses_response_value_b"
label class="radio-label" for="responses_response_value_b">"b"
input class="radio0" name="responses[0][response_value]" type="radio"
      value="c" id="responses_response_value_c"
label class="radio-label" for="responses_response_value_c">"c"
input class="radio0" name="responses[0][response_value]" type="radio"
      value="d" id="responses_response_value_d"
label class="radio-label" for="responses_response_value_d">"d"

And here’s what my params look like in Terminal after submitting the form with multiple responses (3 Responses to 3 Questions):

Parameters: {"utf8"=>"✓", "authenticity_token"=>"...", 
"responses"=> {
     "0"=>{"response_value"=>"a", "user_id"=>"23", "question_number"=>"1"},
     "1"=>{"response_value"=>"b", "user_id"=>"23", "question_number"=>"2"},
     "2"=>{"response_value"=>"c", "user_id"=>"23", "question_number"=>"3"}},
"commit"=>"Submit Exam", "method"=>"post"}

Great! Now some additional nuances. As we mentioned above, on the fields_for call, we pass in include_id: false. This is because, according to the fields_for documentation, it automatically generates a hidden field to store the ID. However, in our case this is interfering with our Responses and the name values linking to the params. To turn this off, we pass in include_id: false to prevent fields_for from rendering it automatically.

A similar thing happens in the collection_radio_buttons code; you see I pass in the include_hidden: false value. This is because Rails 5 by default now creates a hidden value that is blank so that even if the user submits the form without clicking on any radio button, a blank value will be sent to the server (default HTML doesn’t do that). I learned this from this blog post by Prajakta Tambe, which explains this very well and includes other links to the documentation. Since I’m customizing all of these inputs, I needed to turn off that default behavior.

Whew! That was a lot of info. Hopefully it sort of makes sense to you? This took me forever to understand and figure out, and hopefully it saves you time.

Update: Someone asked about how to access these params in the controller, so I’m adding the controller code below. In the create function, basically just iterating through each of the “responses” in the params, and creating a new instance of the Response model for each, passing in the parameters that were sent in the params hash.

def create
  params["responses"].each do |key, value|

Also, as explained in this Vic Friedman post, we have to adjust the strong params function to handle submitting the params in the altered structure in the create function.

So in the bottom of the controller, our new params function now accepts an argument, which is our new params structure:

def response_params(my_params)
  my_params.permit(:question_number, :answer_id, :user_id, :correct,       :response_value)

So that’s what I put in the controller code.


Vic Friedman

Ryan Bates Gist


Rails Documentation for Building Complex Forms

Railsguides – Simple Form Array Text Input

Ryan Bates Advanced Rails Recipes

StackOverflow Question 1

StackOverflow Question 2

Using AJAX and jQuery in Rails 5

RichOnRails – AJAX in Rails