Skip to main content

10 Realtime SSH Logs with ActionCable

Episode 245 · May 23, 2018

Streaming realtime progress to the browser is easy with ActionCable. This episode covers how we send over SSH logs in realtime inside Hatchbox.io

ActionCable


Transcripts

What's up guys? This episode I wanted to talk about a feature of HatchBox that has caught a lot of people's eyes and they have asked me to talk about how it works, so what I'm talking about here in this episode is the real time SSH logs that happen in your browser, so this is pretty straight forward actually but when you click deploy and your app begins deploying, you can see a live output of the logs as it's running, so it's doing all of this and updating these logs and generating all of this output and sending it back over from the server to the HatchBox server to your browser using ActionCable.

Let's write out a quick little diagram showing how all of this works behind the scenes at an architecture level. First you have your browser, visits HatchBox.io over HTTP, then your browser as part of the JavaScript will set up a HatchBox.io websocket connection over ActionCable, and then when you click the deploy button you make a request to HatchBox to tell it to deploy and then HatchBox will start talking to your server over SSH and we're using the NET SSH library that is built into ruby for that, and so that is basically making a TCP connection to the server and says: Hey, I'm going to give you a command, and the server says: Ok, cool, and then starts sending us the output back, so what we can do is capture chunks of that output that it is sending back, and then we can send that over your websocket connection here back to the browser, so then we take a chunk of SSH data, and then we send that from our background worker that's doing all of that NET SSH connection to your server, that chunk of data will then be passed over to Redis so that ActionCable can parse it, and then ActionCable will send that over to your browser, so this is HatchBox with websocket/ActionCable, I'll make this a little bit more clear, ActionCable as well, and then that sends it back to your browser. Now we use this feature actually in a bunch of different places herr so for example if you want to see your rails logs on the server, this is using the exact same feature, basically when you visit this page, we kick off a background job that goes and grabs the last 300 lines of your rails logs and so it runs a really simple command.

tail -n 300 /home/deploy/errbit/current/logs/production.log

This command if you go into your server and run it, it will just print out the last 300 lines of that, so what I'm doing is I create an SSH connection in the background job, to talk to your server just like you do in your terminal, and then I tell it hey run this command just like you do when you type it in, hit enter, and then you immediately get the output in your console, in your terminal and I will get effectively the same thing but I'll get it in chunks of text in the SSH connection which will call a little ruby callback that I say: Hey, go pipe this over to the browser with ActionCable, and that's exactly what we do, so that is all that's really going on here, and we have this interesting modification for the NET SSH connection session, this is the class that is the one actually executing commands on a live SSH session or connection, and so we have this method that I've taken from a Stack Overflow post, that added this exec_sc method and so this will retrive, or it will execute a command and retreive the logs as well as check for exit codes and exit signals which is really important to determine if the command failed or not, so what happens is it creates that channel, and it sits here and loops after it has set up these callbacks, so what happens is it created this SSH connection, it is sending and receiving messages back and forth and doing all of this as part of this loop, and whenever it receives a message of data, it will call this block, and so you're free to do anything you want in there, and so for me, if you passed in a script, I will call concat_log on that script so it knows: Hey, if you gave me a scropt, I will go add this to my logs, and that is where I collect those logs and send them over with the ActionCable, you're free to do anything you want in here, and this standard out is just basically saving all of this in memory so that when the command finishes, you can have the standard out as a whole at the very end but we don't want to wait until the very end, we want every piece of data as soon as we can get it, and so what happens is then we take every piece of data and concatenate it with ActionCable in your browser, and so we're taking every little piece of it that we get as soon as possible and sending it over so it comes across in real time, instead of waiting to the very end and you might run a deply that takes a while because it's got to run migrations and compile assets and all of that stuff, and of course that allows us to make sure that our users aren't confused if things take a while, they actually feel like it's running and has installed or broken in some way. When we call this concat log method on the server side it just calls the ActionCable broadcast to the server channel and we pass in the server, and then on the client side this is basically what we do. Now it needs some refactoring and cleaning up because it's changed a lot over the years, but basically we have a section for processing rails logs, we have a section for processing nginex logs, and SSH logs, and each one of these kind of works a little bit differently, with SSH logs, we want to append to the bottom and then continue scrolling up as we get new information just like your terminal would do, you don't wan to have it stuck at one height and then it keeps printing out things that you can't see, and so that would be something that the SSH logs would have uniquely as a feature and then rails log, if you see the words "No such file or directory" we can probably guess that you were probably using the rails 12 factor gem from Heroku and you haven't output any of those logs to your server, so you need to remove that gem so that we can retrive logs in the future, and then our NgineX logs are handled a little bit differently, and I'll probably go clean this up but for now it's just a few lines of code and it's nothing very complex, but as this grows and it has more features, we might have the ability to turn on or off this autoscrolling or whatever, so we might want to go in here, and refactor that as we add some more features, but that's as simple as it is, we have the NET SSH connection open, it send us some data back like snippets of the output, and we take those snippets of the output, broadcast them over ActionCable and then just insert them onto the page with our JavaScript, there is nothing omre to this, it's just really straightforward.

So that's it for this episode, basically the important piece here was that callback method on data for our NET SSH connection that gives us periodic status updates for the long running job which is really really nice, so anytime that you want to implement something like this to show progress to your users in real time, you're gonna want to do basically the same thing, create some sort of callback that your other code can use to pipe it over ActionCable. Now if you're interested in talking about sockets work, so those connections like the SSH connection, or just your browser connecting to your rails app or your rails app listening for connections, we can talk about sockets in the guture, and build our own web server and all of that good stuff, so if you're interested in that, let me know in the comments below and we'll dive into that in the future if you guys want. Until then I will talk to you guys in the next episode, Peace v

Transcript written by Miguel

Discussion


Gravatar
Definitely interested in sockets :)

Gravatar
I'm very excited to learn more about Socket. 🙂

Gravatar
Thanks Chris!!!

Gravatar
Chris, any chance you could provide the code for this episode? Even if it's just snippets and not complete would be helpful.
Gravatar
https://github.com/search?l=Ruby&q=commandexecutionfailed&type=Code

Login or create an account to join the conversation.