Rails Paperclip Attachments Amazon Cloudfront CDN with Signed URLs using the Cloudfront-signer Gem and Tricky YAML

So here’s what I was trying to do. I have a Rails app where users upload attachments. In my case they are pdf’s, but I think this still applies if they were images or any other file type. I use the paperclip gem to handle the pdf attachments, and store them on Amazon S3 bucket cloud storage. This was all working fine. However, I wanted to make 2 improvements:

1. Use a CDN such as Amazon Cloudfront to deliver these attachments faster globally. See this blog post that first put me on to the idea.
2. I originally made the S3 bucket public to serve these assets, but I wasn’t happy with this. I wanted more security and privacy for these attached files. So after setting up the Amazon Cloudfront CDN, I wanted to use signed and expiring URL’s to protect the content.

While initially this seemed like a somewhat common and straightforward thing to do, I ran into a lot of annoying little hiccups that took way too much time to figure out. I never found somewhere where the steps to do this were all in one place. So I’m writing this to hopefully do that.

First you need to pick your CDN. There are many options, including one that looks promising called Cloudinary (and their Attachinary library), which I might switch to later and do another post or update this one. For now I chose Amazon Cloudfront since it is well documented and has lots of stackoverflow support :).

So to create an Amazon CDN distribution, login to your AWS console and click “Create Distribution”, then select a “web” distribution. For “Origin Domain Name”, pick the S3 bucket where your files are stored. This is the ‘origin’ of the files for Cloudfront. I left the “origin path” blank, as its optional. Next, give it whatever ‘origin-ID’ you want to make it descriptive to yourself. In my case I wanted to select “Restrict Bucket Access”, which then disables the Amazon S3 URLs, so your content can only be viewed from the Cloudfront URL. Then “use an existing identity” for Origin Access Identify if you have one, or create a new one and give it a useful name. Then for simplicity and ease of use I chose “Yes, Update Bucket Policy” under “Grant Read Access”. This allows the Cloudfront distribution to read the files in your S3 bucket so it can distribute them. If you don’t have it update the policy automatically, you will need to manually update the bucket policy on your S3 bucket to something like this:


{
"Version": "2008-10-17",
"Id": "PolicyForCloudFrontPrivateContent",
"Statement": [
{
"Sid": "1",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::cloudfront:user/CloudFront Origin Access Identity xxxx"
},
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::bucketname/*"
}
]
}


I left the “Origin Custom Headers” blank. Setup the “Default Cache Behavior Settings” however you see fit. I chose to redirect HTTP to HTTPS – its best to not use HTTP whenever possible. Select “Yes” for “Restrict Viewer Access” to use signed URLs, and pick the appropriate signers. Everything else I left as the default options. For “Distribution Settings” you can use CNAMEs to set up URLs to use your custom domain, but I didn’t do this. I did choose to use the default Cloudfront SSL Certificate, again for simplicity. Everything else I left as defaults, and you’re all set up! It takes a few minutes for your distribution to be fully enabled after you create it.

Awesome, we now have a Cloudfront distribution setup to deliver the assets from our S3 bucket, making it much faster. And we can use signed URLs to protect the privacy of these files. Let’s do that next.

The basic structure of signed URL’s is as follows:

{base_url}?{query_string_parameters}Expires={expiration_in_seconds_UTC}&Signature={hashed_and_signed_policy_statement}&Key-Pair-Id={key_pair_id}

The example in the AWS documentation is:

http://d111111abcdef8.cloudfront.net/image.jpg?color=red&size=medium&Expires=1357034400&Signature=nitfHRCrtziwO2HwPfWw~yYDhUF5EwRunQA-j19DzZr vDh6hQ73lDx~-ar3UocvvRQVw6EkC~GdpGQyyOSKQim-TxAnW7d8F5Kkai9HVx0FIu- 5jcQb0UEmatEXAMPLE3ReXySpLSMj0yCd3ZAB4UcBCAqEijkytL6f3fVYNGQI6&Key-Pair-Id=APKA9ONS7QCOWEXAMPLE

Ok, so the URL, query parameters, and expiration are easy. But we need to get the Signature and Key-Pair-ID.

Let’s do the Key-Pair-ID first. In your AWS console, go to the Security Credentials section, and open Cloudfront Key Pairs. Create a new key pair, then download the private and public keys. Of course, save the private key in a secure location and don’t share it with anyone. We’ll come back to them in a minute.

Now we need to add Trusted Signers, to give someone the authority to create signed URL’s. For web distributions, you need to create a Trusted Signer based on each cache behavior you have setup. For me, there was just one. Go to your AWS Cloudfront console and select the distribution settings for the distribution you’re working with. Go the Behaviors tab and select the behavior in the list – for me the only was was the redirect HTTP to HTTPS behavior. Select Edit, and choose Restrict Viewer Access, and select yourself or whoever you wish as your Trusted Signer. This account can create signed URLs.

Ok, now that we have Trusted Signers assigned, we can go ahead and create our signed URLs. The Amazon documentation for how to create signed URLs is pretty good, and you can do it manually yourself. I found this gist that pretty much gives you the ruby code to create your signed URLs – its only a few lines. But wait, this is Rails, there’s gotta be a gem for that, right? Yep, there is! It’s called cloudfront-signer.

Install the gem and run the initializer:

bundle exec rails generate cloudfront:install

Now open the created config/initializers/cloudfront_signer.rb file. This is where we will put our credentials for creating the signed URL.

As described above, we need the Cloudfront Key-Pair-ID to append to the end of the URL. This was a bit confusing for me, but your Key-Pair-Id is called “Access Key ID” in your Cloudfront Security Credentials page where you downloaded the private and public keys. See screenshot:

Copy the text string of capital letters under “Access Key ID” (mine is blanked out in the screenshot for security) and paste it into the cloudfront-signer.rb file for “key_pair_id”:

Aws::CF::Signer.configure do |config|
config.key_path = '/path/to/keyfile.pem'
# or config.key = ENV.fetch('PRIVATE_KEY')
config.key_pair_id = 'XXYYZZ' # copied from your Cloudfront "Access Key ID"
config.default_expires = 3600
end

Ok, last thing we need for configuration is your private key that you downloaded in the steps above. You can either save the .pem file itself into your Rails project somewhere – making sure to add it to .gitignore so it doesn’t become public in your version control!! – or you can copy/paste the text of the key into an environment variable. I originally used the first option, but then when I wanted to push to heroku in production, I didn’t know how to reference this file in production since I didn’t have it in version control, so I decided to use an environment variable.

Sounds simple right? Well, usually, but here’s where I ran into more trouble. I didn’t know that actually the \n line breaks in your private key file are critical, and can’t be removed. (I learned that from this unrelated github issue) So to put a multi-line environment variable in an application.yml file (as is done with the figaro gem in Rails) there are some fancy yaml tricks that you need to do. If you don’t you keep getting some weird errors, such as Neither PUB key nor PRIV key: nested asn1 error (OpenSSL::PKey::RSAError) because OpenSSL tried to sign the key without the \n line breaks, making it an invalid key. It took me a long time to debug this error, but hopefully this helps save you time.

Based on what I learned from this StackOverflow question, you need to add |- in front of your key in the application.yml file to keep the line breaks, but not create one at the end of the file. So the code in your application.yml will look like this:

AWS_CLOUDFRONT_PRIVATE_KEY: |-
-----BEGIN RSA PRIVATE KEY-----
#long string key here
-----END RSA PRIVATE KEY-----

That sneaky little |- is critical!

Ok, so we have all our credentials set up. If you’re not using the gem, you need to do a bit more manually. Signed Cloudfront URL’s use a policy statement. You can create a custom one with various parameters, or use a canned policy. I just used a canned policy statement. The example in the AWS documentation is:

{
"Statement":[
{
"Resource":"base URL or stream name",
"Condition":{
"DateLessThan":{
"AWS:EpochTime":ending date and time in Unix time format and UTC
}
}
}
]
}

Our cloudfront-signer gem conveniently creates the policy statement for us.

To use the gem to create a signed URL, just call Aws::CF::Signer.sign_url {your_attachment_url}, expires: Time.now + 600. In my case I put this in my show.html page where I wanted to display the attached pdf in an iframe.

Now one more tricky note was how to add our multiline environment variable to heroku while preserving the line breaks. Based on this StackOverflow question, you can use the cat function in the terminal to save it as a variable in your terminal session, then use that to create the variable on heroku:

testvar=$(cat myfile.txt). Then heroku config:add EC2_PRIVATE_KEY="$testvar"

Whew, more complicated than it sounded at the start, huh? Hopefully this was useful and saved you some headaches!

Here’s a list of helpful links:

http://www.akitaonrails.com/2016/07/28/updating-my-old-posts-on-uploads

https://devcenter.heroku.com/articles/using-amazon-cloudfront-cdn

https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-creating-signed-url-canned-policy.html#private-content-creating-signed-url-canned-policy-procedure

https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/PrivateContent.html

https://stackoverflow.com/questions/3790454/in-yaml-how-do-i-break-a-string-over-multiple-lines

https://stackoverflow.com/questions/6942600/multi-line-config-variables-in-heroku

https://github.com/chef-cookbooks/jenkins/issues/148

https://stackoverflow.com/questions/43082918/how-to-sett-multiline-rsa-private-key-environment-variable-for-aws-elastic-beans

https://stackoverflow.com/questions/14391312/openssl-neither-pub-key-nor-priv-key-nested-asn1-error

https://stackoverflow.com/questions/2632457/create-signed-urls-for-cloudfront-with-ruby

https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-restricting-access-to-s3.html#private-content-creating-oai-console

http://www.akitaonrails.com/2017/06/28/rails-5-1-heroku-deployment-checklist-for-heroku

http://www.akitaonrails.com/2016/07/28/updating-my-old-posts-on-uploads

http://www.akitaonrails.com/2017/07/07/upcoming-built-in-upload-solution-for-rails-5-2-activestorage

https://github.com/thoughtbot/paperclip/wiki/Restricting-Access-to-Objects-Stored-on-Amazon-S3

https://stackoverflow.com/questions/4003828/aws-s3-ruby-on-rails-heroku-security-hole-in-my-app

https://stackoverflow.com/questions/3897837/rails-3-paperclip-s3-howto-store-for-an-instance-and-protect-access

https://stackoverflow.com/questions/41926167/rails-open-uri-and-amazon-s3-open-http-403-forbidden-openurihttperror

https://stackoverflow.com/questions/19176926/how-to-make-all-objects-in-aws-s3-bucket-public-by-default

Add Amazon Cloudfront as CDN from S3 Storage

I have a small app that was serving file attachments directly from S3. Although I wasn’t doing any processing, it’s still preferred to deliver those over some type of CDN. Here I’ll lay out how I added an Amazon Cloudfront CDN so now my S3 assets are served from there instead of directly from S3.

I’m assuming you already have S3 setup to store your attachment files. This post will talk about the setup using the Paperclip gem, but you can use others like Shrine or Cloudinary.

So first you need to setup an Amazon Cloudfront distribution that will be linked to your S3 bucket.

Go to your Amazon AWS console and create a new Cloudfront distribution. There are loads of settings, and most of them I just left as the default. For the first one “Origin domain name”, click in the box and it will give you the list of all the S3 buckets in your account. Select the one you want. It will auto-fill the Origin-ID box. You can leave the Origin Path blank, or fill it in with a specific folder in your bucket that you want to use. For Protocol Policy, I switched it to redirect HTTP to HTTPS to ensure everything is using HTTPS. Everything else I left as the default. You can change them later. Your distribution is now created. It will take a few minutes to be deployed.

Note that your file needs to be set to public access.

To test it, go to your Amazon S3 bucket, and get the link to a file. It will look like s3.amazonaws.com/bucketname/path/to/filename.jpg. Open it in the browser. Now go to your cloudfront distribution, and replace “s3.amazonaws.com/bucketname/” with your cloudfront url, “xxxx.cloudfront.net”. You should be able to load the file in the browser from the cloudfront url.

What was not obvious to me is that when using the cloudfront url, you don’t need the bucket name. So for the S3 URL of

https://s3.amazonaws.com/bucketname/path/to/filename.jpg

the Cloudfront url is

https://xxxx.cloudfront.net/path/to/filename.jpg

No we can go to our Rails app and setup Paperclip to deliver the attachments from Cloudfront. In config/environments/production.rb

config.paperclip_defaults = {
:storage => :s3,
:url => ':s3_alias_url',
:s3_host_alias => "d30l9ueliue92c.cloudfront.net",
:path => '/:class/:attachment/:id_partition/:style/:filename',
:s3_credentials => {
:bucket => ENV['AWS_BUCKET'],
:access_key_id => ENV['AWS_ACCESS_KEY_ID'],
:secret_access_key => ENV['AWS_SECRET_ACCESS_KEY']
},
:s3_region => ENV['AWS_REGION'],
:endpoint => ENV['AWS_ENDPOINT'],
:s3_protocol => 'https',
}

With the key being the :s3_host_alias and :url and :path settings.

And you’re all set! You can also set up Cloudfront to deliver your actual Rails app assets (javascript and css), but I’ll do that in a later post.

credit here

Setting Up Email in a Rails 4 app with Action Mailer in development and SendGrid in production using Heroku

Here is a demo of how I created an email w/ user-mailer in rails

First off, from the Rails Guide generate a UserMailer

Then in the user_mailer.rb create a function that sends the email

Pretty simple right?  Now we need to create the view.  This is where you can design the email and whatever text you want to put in it. Create a file in the app/views/user_mailer folder.  The name of the file should match the name of the function you defined in the user_mailer.rb file.  Mine is welcome_email.html.erb

Now that we have the function to send the email, and the text of the email itself,  we just need to add a line in the controller to have it send the email when we want it to.

You can call the mailer function wherever you want to send the email, in this case we can just send it when a new user signs up

Now we need to set up the config files for the email address we will use to send emails. For development, in the config/environment/development.rb file:

Set the config.action_mailer.default_url_options to ‘localhost:3000’ in development, or whatever port you are using to test.  In production, you would set it to the url of your production app.  Then set up the email smtp settings according to your email provider.  I’ve shown the settings for gmail here.

So great, now we have it set up so a new user will receive an email, at least in development.  Now getting this working in production can be slightly harder.  But I’m going to walk through one super easy solution of how to set this up using Heroku and SendGrid, but there are probably a million different ways to do this depending on your hosting, your email, etc.

So the first thing to do before moving to production is to protect the sensitive email sign in information that we just put in the config/environment/development.rb.  The easiest way to do this is to use environment variables, and a simple way to do that is to use the figaro gem.  I wrote a quick post on how to setup environment variables with the figaro gem here, so take a minute and set that up in your app first.  This will keep your email login info from getting pushed to github, in case you are using a public repository.  It should then look something like this:

So, assuming you have your app pushed to heroku for production, the next thing we need to do is setup an email add-on for your heroku app.  Of course you could fully create your own email solution, but if you know how to do that then you wouldn’t be reading this blog, would you? So, for a simpler solution, first you can choose one of the email related add-ons from the heroku add-ons page.  I happen to use SendGrid as I think its one of the simplest to set up and use, so that’s what I’ll go through here.

Add SendGrid to your heroku app by running

heroku addons:add sendgrid:starter

Now SendGrid will have automatically generated a username and password that you can see by running

heroku config:get SENDGRID_USERNAME
heroku config:get SENDGRID_PASSWORD

or simply running heroku config will show you all the heroku config variables.

Next, go to config/environment/production.rb and add our email settings there:

So you can see that SendGrid has set up environment variables on heroku for us, giving us the SENDGRID_USERNAME and SENDGRID_PASSWORD.  This is the same idea that we just did in development using the figaro gem.

Ok, so now push to heroku, migrate the database, and see if the mailer is working in production.  Not too difficult right?  Stay tuned for a follow up post where I’ll describe how to send email asynchronously with sidekiq and redis.

Making text and its parent item with different opacity using bootstrap

I’m using the ‘well’ class from bootstrap.  Since in this case I have a background image behind the well, I wanted to make it somewhat transparent.  But the text in the well becomes really hard to see:

So I wanted to keep the text and buttons inside the well with opacity at 1.0.  How can I do this? Well first, instead of using the opacity value in the css file, us the background-color with its rgba values (a meaning the alpha value, or transparency), like this:

.well {
background-color: rgba(39, 174, 96, 0.6);
}


Then, I put everything inside the well in another class. I called it field

.field {
opacity: 1.0;
}


And now the well stays transparent, but everything else inside is not transparent.

Also, if your changes to the css don’t seem to be showing up in your rails app, don’t forget to precompile your assets:

rake assets:precompile

or

rake assets:precompile RAILS_ENV=production for heroku or your production environment.

Some credit goes to this site.

How to reset the postgres database on heroku

So I’m sure your not like me and you never have to rename your models in your database, because you always plan enough ahead that you name everything perfectly on the first migration.  In case you don’t you may end up renaming tables, adding/removing database columns etc.

My issue came from a join model between Students and Courses that I called Registrations.  Which seemed to be one of the rare occasions where I actually found a word for the join model that made perfect sense in the context and perfect sense in English.  However the issue arose when I then started to use the Devise gem for user authentication.  Confusion and chaos ensued due to the fact that devise uses ‘registrations’ to describe users signing up for your site.  These two uses of ‘registration’ caused overlap in routes and other errors.  So in short, I unfortunately needed to rename the full Registrations scaffold, along with its views etc.  Basically I went through my whole app manually in Sublime Text and replaced all the instances of the registration (except the devise ones of course) with my new model name – attendance.  Of course you also have to rename all the files too.  This is tedious and error prone, and if anyone knows of a better way to do it, please let me know.  Maybe I’ll make one in my free time someday.  This Stack Overflow question goes into a bit more detail of exactly how to rename a full scaffold.

But my point here is heroku.  After going through all the work of renaming the full scaffold and all its files, I ran rake db:reset to reset the database, and then ran the migrations again.  Worked great.  However for some strange reason, this does not work on heroku.  I ran heroku run rake db:reset and kept getting this strange error:

FATAL: permission denied for database "postgres" DETAIL: User does not have CONNECT privilege.

which really confused me as of course I’ve been changing the heroku database all along, why would I suddenly not have privileges?  My first thought was to try running sudo, but after Googling, I came across this SO question with the answer:  On heroku you must run heroku pg:reset – notice the omission of run rake.

Hope this saves you some time!