Alex Deering

Starting a new project and trying to figure out how to approach the main concept so as to pick the right resources to support it. The site is going to be a dynamic group note taking app and has some key features that I think will make one database easier then the other.

• User created note templates (similar to how you can create templates with google docs) that will have set fields and spots to input data. Not entirely sure how to save the templates though, as HTML strings? Json layouts?
• Group editable notes, want note boxes in the templates where any invited user can edit it freely, assuming action cable will be used for this one.

So database is the big choice I think from the get go. Since the note pages are dynamic (depending on the template they create) and wont have a set structure, im thinking a nosql option like Firebase cloud firestore would be better. But I know rails tends to prefer a sql based database.

yup, thats my next step and ill post an update here once i have some metrics and can compare

Well was able to easily swap over to DigitalOcean Spaces and will see how that goes on cost. And will start looking into using cloudflares free CDN service. Thanks for the advice and info, youre awesome with helping

Ok, so I'll definitely start looking into switching over to DO Spaces then. I saw digital ocean has a guide for transfering specifically from S3, so will need to check that out and see if I can get swapped over. Not too worried about new images, as right now Im the main person uploading recipes so I just wont upload anything. For the endpoint update, dont need to do any changes with the database info correct? Just update my config files to use the new digitalocean spaces endpoint and it pulls from that and there isnt anything saved in the database expect the filename correct?

I currently am using the ActiveStorage variant features to resize the images to the size I need and using those for display. I have also come across this guide for using DO and cloudfare CDN with activestorage. Looks like they override the public url function to use the CDN domain. So Ill give that a try also

https://vitobotta.com/2019/11/17/rails-active-storage-permanent-urls-with-no-redirects-digital-ocean-spaces-cloudflare/

I have and its the data transfer out for the S3 thats breaking past the free tier and costing all the money. Specifically its all under the bandwidth cost. Apparently Im transfering 2TB worth pictures out a month with about 3k visitors a day. Its $0.09/GB to transfer out.

Im using digital ocean for my servers, so I was considering transfering to their spaces, but not entirely sure how I would go about migrating over and transfering all the current images and active storage references to the new storage. Or if that would fix the issue totally. I know digital ocean is cheaper for storage though so definitely looking into that.

Have had my website up for about a month and a half now and have run into an issue where the data transfer cost for AWS S3 is crazy high (around $200/month). The website is a recipe sharing website so there are a lot of pictures being uploaded and loaded when people search and view the website but this still seems excessively high. Im using default configuration for active storage with some resizing for the different views that the images are display in.

Is there a setting or something that I mightve overlooked to reduce the costs or is this pretty standard to expect for costs? For reference im getting about 3k visitors a day.

Ive looked into trying to setup amazon cloudfront but havent been able to figure out how to set that up with AWS S3 and active storage so the active storage link uses the cloudfront url instead of the S3 url. Would that significantly decrease my costs?

Ok thanks, thats what it was looking like just checking that I was understanding things correctly

Getting around to setting up marketing emails for my site (using jumpstart) and noticed that the marketing integrations (Mailchimp, Convertkit, Drip, etc) are not the same as the transactional email providers. Typically do you use a different provided for the transactional emails like, password reset, welcome emails, etc. and the marketing emails?

Just learning about this part of website management so trying to figure out how all this works and what accounts to set up. Thanks for any info on how email stuff works

This is probably a super easy solution, but for some reasonits eluding me. Im using elasticsearch and searchjoy and passing the searchid so I can track conversions to refine search results. However, I would like to not include the search id in the url in the browser and just have it in the background. Here is my link code:

<%= link_to "View", @recipes.search.present? ? recipe_path(recipe, search_id: @recipes.search.id) : recipe, class: "btn btn-secondary btn-small justify-center w-1/2 self-center mt-2", data: {action: "click->modal#linkClose"} %>

So how do I change the activestorage url to use the CF url?

Ok, ill give it a shot. So shouldnt need to change anything in my code and just set up the cloudfront in AWS

Im trying to setup cloudfront for my aws s3 storage of user uploaded images. Not sure what settings to change in the storage.yml file to point to the cloudfront and still work with everything else. Would it be just changing the endpoint variable to the cloudfront url?

How would we go about setting this up to work for our S3 bucket and active storage. Im assuming would need to adjust the storage.yml file to point to the Cloudfront domain right? But would that mess up the the upload with activestorage?

In the storage.yml can you just add in the endpoint to the amazon configuration like this:

amazon:
  service: S3
  access_key_id: <%= Rails.application.credentials.dig(:aws, :access_key_id) %>
  secret_access_key: <%= Rails.application.credentials.dig(:aws, :secret_access_key) %>
  region: us-west-2
  bucket: living-recipe-images
    endpoint: 'abcdefg12345678.cloudfront.net'

Nice, glad you were able to figure that out. Hatchbox is amazing so glad I could help with something, even if its just pointing you in the direction of the problem.

Just an update for you. I ended up switching to a managed elasticsearch (Elastic Cloud/AWS) since Im not super familiar with ES either and could get it up and running for now.

As for the issue, it mightve been the server size. I was using a $20 (4gb, 2CPU) server from DigitalOcean for only the ES service. Some of the responses I was finding while troubleshooting on stackoverflow was that if there isnt enough memory when creating it can fail starting up. Might be worth testing with different size Servers and seeing if a beefier server allows it to start up properly.

Also I did SSH into the server and uninstalled ES and reinstalled it following the same procedures I did on local dev machine and got the same UUID error. Which is what eventually led me to think it might be a resource issue with the server. Hope this helps with any troubleshooting you do with your deploy scripts.

Hatchbox is awesome and has saved me a lot of work btw. Living-recipe.com is up and testing quickly thanks to your services. Thanks again for the help

Beautiful, that was it. Thanks so much

Ive noticed that anywhere that I adjust the css classes dynamically (select2 boxes and one of my stimulus controllers adds a border when a div is clicked) the classes are added but the display isnt updated on the webpage. Not sure what would cause this, Ive inspected the page and confirmed that the javascript is running, there are no errors and the css classes are added to the div, but just not displaying.

Heres a picture of the select2 boxes. You can see the UL html was added, and the select fields have the hidden property, but are still showing

https://imgur.com/a/i0UCbVD

Just an update, after uninstalling and reinstalling elasticsearch, still getting the same issue. For now Ive moved to a managed service (Elastic Cloud) and its working. So pretty sure it was something to do with the install on the server. I will reattack at a later date

Tested the connection with both the deployed and my development machine and compared the results:
Deployed:

{
  "name" : "living-recipe-elasticsearch",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "_na_",
  "version" : {
    "number" : "7.5.1",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "3ae9ac9a93c95bd0cdc054951cf95d88e1e18d96",
    "build_date" : "2019-12-16T22:57:37.835892Z",
    "build_snapshot" : false,
    "lucene_version" : "8.3.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

Localhost:

{
  "name" : "ubuntu",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "uA-3VHNeQ5mzozcnB8R-ww",
  "version" : {
    "number" : "7.5.1",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "3ae9ac9a93c95bd0cdc054951cf95d88e1e18d96",
    "build_date" : "2019-12-16T22:57:37.835892Z",
    "build_snapshot" : false,
    "lucene_version" : "8.3.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

Everything is the same except for the cluster_uuid is "na" for the deployed version

Ok, this is what I get with that:

I, [2020-01-02T05:20:21.550200 #8714]  INFO -- : [e05340b7-b4d6-4e1d-bedd-11b48ee75d62] Started GET "/" for 199.127.56.90 at 2020-01-02 05:20:21 +0000
D, [2020-01-02T05:20:21.652893 #8714] DEBUG -- : [e05340b7-b4d6-4e1d-bedd-11b48ee75d62]   User Load (1.9ms)  SELECT "users".* FROM "users" WHERE "users"."id" = $1 ORDER BY "users"."id" ASC LIMIT $2  [["id", 1], ["LIMIT", 1]]
I, [2020-01-02T05:20:21.698751 #8714]  INFO -- : [e05340b7-b4d6-4e1d-bedd-11b48ee75d62] Processing by RecipesController#index as HTML
D, [2020-01-02T05:20:21.725351 #8714] DEBUG -- : [e05340b7-b4d6-4e1d-bedd-11b48ee75d62]    (5.9ms)  SELECT "saved_recipes"."recipe_id" FROM "saved_recipes" WHERE "saved_recipes"."user_id" = $1  [["user_id", 1]]
D, [2020-01-02T05:20:21.734363 #8714] DEBUG -- : [e05340b7-b4d6-4e1d-bedd-11b48ee75d62]    (4.4ms)  SELECT DISTINCT "taggings"."tag_id" FROM "taggings" WHERE 1=0
D, [2020-01-02T05:20:21.745764 #8714] DEBUG -- : [e05340b7-b4d6-4e1d-bedd-11b48ee75d62]    (1.7ms)  SELECT COUNT(*) AS count_all, "taggings"."tag_id" AS taggings_tag_id FROM "taggings" WHERE 1=0 GROUP BY "taggings"."tag_id"
D, [2020-01-02T05:20:21.765707 #8714] DEBUG -- : [e05340b7-b4d6-4e1d-bedd-11b48ee75d62]   Recipe Search (18.9ms)  curl http://10.138.104.120:9200/recipes_production/_search?pretty -H 'Content-Type: application/json' -d '{"aggs":{"course_name":{"filter":{"bool":{"must":[{"term":{"approved":true}},{"term":{"master":true}}]}},"aggs":{"course_name":{"terms":{"field":"course_name","size":1000,"order":{"_key":"asc"}}}}},"cuisines":{"filter":{"bool":{"must":[{"term":{"approved":true}},{"term":{"master":true}}]}},"aggs":{"cuisines":{"terms":{"field":"cuisines","size":1000,"order":{"_count":"desc"}}}}},"techniques":{"filter":{"bool":{"must":[{"term":{"approved":true}},{"term":{"master":true}}]}},"aggs":{"techniques":{"terms":{"field":"techniques","size":1000,"order":{"_count":"desc"}}}}}},"query":{"bool":{"must":{"match_all":{}},"filter":[{"term":{"approved":true}},{"term":{"master":true}}]}},"sort":{"_score":"desc"},"timeout":"61s","_source":false,"size":30}'
I, [2020-01-02T05:20:21.766187 #8714]  INFO -- : [e05340b7-b4d6-4e1d-bedd-11b48ee75d62] Completed 500 Internal Server Error in 67ms (Searchkick: 18.9ms | ActiveRecord: 24.4ms | Allocations: 6790)
F, [2020-01-02T05:20:21.766946 #8714] FATAL -- : [e05340b7-b4d6-4e1d-bedd-11b48ee75d62]   
[e05340b7-b4d6-4e1d-bedd-11b48ee75d62] Elasticsearch::Transport::Transport::Errors::ServiceUnavailable ([503] {"error":{"root_cause":[{"type":"cluster_block_exception","reason":"blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];"}],"type":"cluster_block_exception","reason":"blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];"},"status":503}):
[e05340b7-b4d6-4e1d-bedd-11b48ee75d62]   
[e05340b7-b4d6-4e1d-bedd-11b48ee75d62] app/helpers/recipe_search.rb:23:in `search'
[e05340b7-b4d6-4e1d-bedd-11b48ee75d62] app/controllers/recipes_controller.rb:18:in `index'
I, [2020-01-02T05:20:21.767808 #8714]  INFO -- : [e05340b7-b4d6-4e1d-bedd-11b48ee75d62] Processing by ErrorsController#internal_server_error as HTML
I, [2020-01-02T05:20:21.770647 #8714]  INFO -- : [e05340b7-b4d6-4e1d-bedd-11b48ee75d62]   Rendering errors/internal_server_error.html.erb within layouts/errors
I, [2020-01-02T05:20:21.771662 #8714]  INFO -- : [e05340b7-b4d6-4e1d-bedd-11b48ee75d62]   Rendered errors/internal_server_error.html.erb within layouts/errors (Duration: 0.9ms | Allocations: 225)
I, [2020-01-02T05:20:21.774468 #8714]  INFO -- : [e05340b7-b4d6-4e1d-bedd-11b48ee75d62]   Rendered shared/_favicons.html.erb (Duration: 0.4ms | Allocations: 127)
I, [2020-01-02T05:20:21.774965 #8714]  INFO -- : [e05340b7-b4d6-4e1d-bedd-11b48ee75d62] Completed 500 Internal Server Error in 7ms (Views: 6.3ms | ActiveRecord: 0.0ms | Allocations: 2620)

So it is going to the right url.