FreeCodeCamp and Learning Node: HTTP File Server

Last time we learned how to return a time in response to a request.  Now we’re going to learn to return a file in response to a request.  We’re coming towards the end of the Node tutorials and so things are getting more interesting and complex.  This exercise was an interesting one; we’ll be serving files as part of the challenges for FreeCodeCamp in the future…

For now though, let’s look at  the Directions and Hints:

 ## HTTP FILE SERVER (Exercise 11 of 13)
  Write an HTTP server that serves the same text file for each request it receives.
  Your server should listen on the port provided by the first argument to your program.
  You will be provided with the location of the file to serve as the second command-line argument. You must use the fs.createReadStream() method to stream the file contents to the response.
 ─────────────────────────────────────────────────────────────────────────────
 ## HINTS
  Because we need to create an HTTP server for this exercise rather than a generic TCP server, we should use the http module from Node core. Like the net module, http also has a method named http.createServer() but this one creates a server that can talk HTTP.
  http.createServer() takes a callback that is called once for each connection received by your server. The callback function has the signature:
     function callback (request, response) { /* … */ }
  Where the two arguments are objects representing the HTTP request and the corresponding response for this request. request is used to fetch properties, such as the header and query-string from the request while response is for sending data to the client, both headers and body.
  Both request and response are also Node streams! Which means that you can use the streaming abstractions to send and receive data if they suit your use-case.
  http.createServer() also returns an instance of your server. You must call server.listen(portNumber) to start listening on a particular port.
  A typical Node HTTP server looks like this:
     var http = require(‘http’)
     var server = http.createServer(function (req, res) {
       // request handling logic…
     })
     server.listen(8000)
  Documentation on the http module can be found by pointing your browser here:
  file:///home/ubuntu/.nvm/versions/node/v4.4.3/lib/node_modules/learnyounode/node_apidoc/http.html

  The fs core module also has some streaming APIs for files. You will need to use the fs.createReadStream() method to create a stream representing the file you are given as a command-line argument. The method returns a stream object which you can use src.pipe(dst) to pipe the data from the src stream to the dst stream. In this way you can connect a filesystem stream with an HTTP response stream.

I admit that at first the directions and hints were a little cryptic, but this exercise turned out to be much easier than I thought.

First, scanning through the material above, we’re going to need http and fs, so we write the require statements for those.  Then, we note that it states that the port and the file path will be provided as command line arguments in our old friend process.argv.  So, as in the past many exercises, we assign those to variables.  That wraps up assignments, and also the easy part.

Next, they give us the skeleton of the server:

</div>
<div> var server = http.createServer(function (req, res) {</div>
<div>       // request handling logic...</div>
<div>     })</div>
<div>     server.listen(8000)</div>
<div>

8000 will of course be replaced by our port, and so our program now looks like this so far:

</div>
<div>var http = require('http');</div>
<div>var fs = require('fs');</div>
<div>var port = process.argv[2];</div>
<div>var filePath = process.argv[3];</div>
<div></div>
<div>var server = http.createServer(function (req, res) {</div>
<div>       // request handling logic...</div>
<div>})</div>
<div>server.listen(port);</div>
<div>

Now we need to write the “request handling logic” they have in there.  We know, from the directions and hints, that we need fs.createReadStream(), and that we’ll also need a src.pipe(dst).  The info on createReadStream is here:

https://nodejs.org/dist/latest-v6.x/docs/api/fs.html#fs_fs_createreadstream_path_options

So, what goes in the parentheses (the parameters) is the file path, which we have a variable for called filePath.  That’s the SOURCE, so we’ll assign it to a variable called src, thus:

</div>
<div>var src = fs.createReadStream(filePath);</div>
<div>

Now, finally, we have src.pipe(dst).  src is the SOURCE (see what I did there?) and dst is the stream we are sending it to.  Here’s where it gets a little tricky.  What are we sending it to?  Well, in the hints we are told, “Both request and response are also Node streams!”  Ahhh, so we want to use the response stream!  So:

</div>
<div>src.pipe(res);</div>
<div>

And…..we pass!  Here’s the final program:

</div>
<div>var http = require('http');</div>
<div>var fs = require('fs');</div>
<div>var port = process.argv[2];</div>
<div>var filePath = process.argv[3];</div>
<div></div>
<div>var server = http.createServer(function (req, res) {</div>
<div>       // request handling logic...</div>
<div>       var src = fs.createReadStream(filePath);</div>
<div>       src.pipe(res);</div>
<div></div>
<div></div>
<div>});</div>
<div>server.listen(port);</div>
<div>

See you next time!

FreeCodeCamp and Learning Node: Time Server

So our next challenge is to build a Time Server.  This one wasn’t too bad.  The only tricky part for me was using the outside library, namely the formatting of the date.

Here are the Directions and Hints:

TIME SERVER (Exercise 10 of 13)
  Write a TCP time server!
  Your server should listen to TCP connections on the port provided by the first argument to your program. For each connection you must write the current date & 24 hour time in the format:
     “YYYY-MM-DD hh:mm”
  followed by a newline character. Month, day, hour and minute must be zero-filled to 2 integers. For example:
     “2013-07-06 17:42”
  After sending the string, close the connection.
 ─────────────────────────────────────────────────────────────────────────────
 ## HINTS
  For this exercise we’ll be creating a raw TCP server. There’s no HTTP involved here so we need to use the net module from Node core which has all the basic networking functions.
  The net module has a method named net.createServer() that takes a function. The function that you need to pass to net.createServer() is a connection listener that is called more than once. Every connection received by your server triggers another call to the listener. The listener function has the signature:
     function listener(socket) { /* … */ }
  net.createServer() also returns an instance of your server. You must call server.listen(portNumber) to start listening on a particular port.
  A typical Node TCP server looks like this:
     var net = require(‘net’)
     var server = net.createServer(function (socket) {
       // socket handling logic
     })
     server.listen(8000)
  Remember to use the port number supplied to you as the first command-line argument.
  The socket object contains a lot of meta-data regarding the connection, but it is also a Node duplex Stream, in that it can be both read from, and written to. For this exercise we only need to write data and then close the socket.
  Use socket.write(data) to write data to the socket and socket.end() to close the socket. Alternatively, the .end() method also takes a data object so you can simplify to just: socket.end(data).
  Documentation on the net module can be found by pointing your browser here:
  file:///home/ubuntu/.nvm/versions/node/v4.4.3/lib/node_modules/learnyounode/node_apidoc/net.html
  To create the date, you’ll need to create a custom format from a new Date() object. The methods that will be useful are:
     date.getFullYear()
     date.getMonth()     // starts at 0
     date.getDate()      // returns the day of month
     date.getHours()
     date.getMinutes()
  Or, if you want to be adventurous, use the strftime package from npm. The strftime(fmt, date) function takes date formats just like the unix date command. You can read more about strftime at:

  [https://github.com/samsonjs/strftime] ( https://github.com/samsonjs/strftime )

The actual documentation link is:

https://nodejs.org/dist/latest-v6.x/docs/api/net.html

Okay, let’s get rolling. Since I decided to be adventurous, we need to install strftime.  We get:

$ npm install strftime
npm WARN package.json chat-example@0.0.0 No repository field.
npm WARN package.json chat-example@0.0.0 No license field.

strftime@0.9.2 node_modules/strftime

On to the actual program. First, I took the skeleton they give us above:

</pre>
</div>
<div>var net = require('net')</div>
<div>     var server = net.createServer(function (socket) {</div>
<div>       // socket handling logic</div>
<div>     })</div>
<div>     server.listen(8000)</div>
<div>

Now, 8000 isn’t going to necessarily be the socket we want.  We need to use the argument supplied to us.  As always, that’s the third element in process.argv (even though it says first in the hints – I think they mean it’s the first usable argument).  I added that and ran it and the terminal hung up; I figured out that it was still listening because I never told it to stop.  So, I added socket.end() inside var server (where it says “socket handling logic” was a hint to put it there) and I added a console.log to print out the port number to see if I had a working program so far.

Hey, build a little, test a little.  That was one of the mottoes of the moon program and NASA back in the 60s, and it works for me!

 

Here’s what I had so far:

</pre>
</div>
<div>var net = require('net');</div>
<div>var strftime = require('strftime');</div>
<div>var port = process.argv[2];</div>
<div></div>
<div>var server = net.createServer(function (socket) {</div>
<div>        console.log(port);</div>
<div>       // socket handling logic</div>
<div>       socket.end();</div>
<div>     });</div>
<div>     server.listen(port);</div>
<div>

That gives us the output I was looking for.  Obviously, we want our program to do a little more than that, so let’s have a look at the documentation for strftime using the link they gave us.  Note first, of course, that we need to use a require statement.  Next, we see some examples.

</pre>
</div>
<div>console.log(strftime('%B %d, %Y %H:%M:%S')), which returns April 28, 2011 18:21:08</div>
<div></div>
<div>console.log(strftime('%F %T', new Date(1307472705067))), which returns 2011-06-07 18:51:45</div>
<div>

Now, we’re going to need today’s date, so we add a variable for it using the JavaScript Date.now() function.  We want to printout the date and the time, less the seconds.  Returning to the documentation for strftime, we can see that to do that, we can use the first part of the second example, ‘%F’, and most of the last part of the first example, %H:%M. So, now I had:

</pre>
</div>
<div>var net = require('net');</div>
<div>var strftime = require('strftime');</div>
<div>var port = process.argv[2];</div>
<div></div>
<div>var today = Date.now();</div>
<div></div>
<div>var server = net.createServer(function (socket) {</div>
<div>        var data = (strftime('%F %H:%M', new Date(today)));</div>
<div></div>
<div>       // socket handling logic</div>
<div>       socket.end(data);</div>
<div>     });</div>
<div>     server.listen(port);</div>
<div>

This prints the date out correctly, but still doesn’t pass!!!  What???

I got this back:

Your submission results compared to the expected:
                 ACTUAL                                 EXPECTED
────────────────────────────────────────────────────────────────────────────────
   “2016-05-21 16:21”                  ==    “2016-05-21 16:21”
                                                 !=    “”
────────────────────────────────────────────────────────────────────────────────

  ✗  Submission results did not match expected!

Now, what was going on?

Simple: the directions specify that the date must be followed by a newline character.  I added it and lo and behold, my program passed!

</pre>
</div>
<div>var net = require('net');</div>
<div>var strftime = require('strftime');</div>
<div>var port = process.argv[2];</div>
<div></div>
<div>var today = Date.now();</div>
<div></div>
<div>var server = net.createServer(function (socket) {</div>
<div>        var data = (strftime('%F %H:%M', new Date(today))) + '\n';</div>
<div></div>
<div>       // socket handling logic</div>
<div>       socket.end(data);</div>
<div>     });</div>
<div>     server.listen(port);</div>
<div>

And there we go!  See you next time!

FreeCodeCamp and Learning Node: Juggling ASync

On to the next exercise in the FreeCodeCamp learnyounode sequence!  The processes are asynchronous, but we want to tame them so they get us our data in order.  Let’s look at the directions and hints:

JUGGLING ASYNC (Exercise 9 of 13)

  This problem is the same as the previous problem (HTTP COLLECT) in that you need to use http.get(). However, this time you will be provided with three URLs as the first three command-line arguments.
  You must collect the complete content provided to you by each of the URLs and print it to the console (stdout). You don’t need to print out the length, just the data as a String; one line per URL. The catch is that you must print them out in the same order as the URLs are provided to you as command-line arguments.
 ─────────────────────────────────────────────────────────────────────────────
 ## HINTS
  Don’t expect these three servers to play nicely! They are not going to give you complete responses in the order you hope, so you can’t naively just print the output as you get it because they will be out of order.
  You will need to queue the results and keep track of how many of the URLs have returned their entire contents. Only once you have them all, you can print the data to the console.
  Counting callbacks is one of the fundamental ways of managing async in Node. Rather than doing it yourself, you may find it more convenient to rely on a third-party library such as [async]( http://npm.im/async ) or [after]( http://npm.im/after ). But for this exercise, try and do it without any external helper library.

Okay, as always, We want to declare our variables first.  We’re going to need http, since we’re using http.get().  We’re also going to want bl, since we are using buffer list for the contents of the URLs, as we did in the previous exercise.  We’re also going to use the skeleton of the call to http.get() that we used in the last exercise:

var bl = require('bl');
var http = require('http');

http.get(addr, function callback(response) {

response.pipe(bl(function (err, data) {
if(err)
console.log(err);
}))

});

Okay, so if we run this now, we get back… a mess.  Remember, there are THREE inputs coming that we want to process one after the other – this suggests a loop.

So, let’s put in a loop:

for(var count = 0; count < process.argv.length; count++) {

http.get(process.argv[count], function callback(response) {

response.pipe(bl(function (err, data) {
if(err) {
console.log(err);

console.log(data.length);
console.log(data.toString());
}));

});
};

Umm…ewww.  No, that was no good.  I messed around with this for awhile, and still didn’t get what I wanted.  At one point, I came close – getting the results back, but not in order.  So, I did some web surfing, as suggested by FreeCodeCamp (you remember how they always say to use Read-Search-Ask, right?), and found this very helpful link on “Four ways of handling asynchronous operations in node.js“.  If you look about halfway down the page, you’ll see a section titled “Reading the files sequentially using callbacks”.

 

Now, here the poster is reading the contents of files, and not URLs.  He is also using fs.readdir.  But the idea is the same.

So, we create a function that contains our http.get() called function readURL().  We need to make it recursive, so it goes through the items provided to us by the arguments.  Keep in mind that, as always with Node, we start with the third element (element[2] since the array starts at 0).  So count’s initial value is 2.

In the response.pipe() statement, we will print the data out – as a string, as mentioned in the directions – and we add that statement.  Also, readURL() is recursive, so if it is successful, we want to call it again, only incrementing count by 1 so that it calls the next argument in process.argv.  If the next argument doesn’t exist, the process terminates and we’re finished.  So, now we have:

var bl = require('bl');
var http = require('http');
var count = 2;

function readURL(count) {

http.get(process.argv[count], function callback(response) {

response.pipe(bl(function (err, data) {
if(err) {
console.log(err);
} else {
console.log(data.toString());
readURL(count + 1);
}
}));

});
}

Let’s run it.

I bet you’re not surprised it doesn’t work, right?  Do you know why?

Because we created a function called readURL()…but we never called it.  We need to add that to the VERY bottom and try again.

 

It worked! And now we can move on to the next exercise.  Here’s the final program:

var bl = require('bl');
var http = require('http');
var count = 2;

function readURL(count) {

http.get(process.argv[count], function callback(response) {

response.pipe(bl(function (err, data) {
if(err) {
console.log(err);
} else {
console.log(data.toString());
readURL(count + 1);
}
}));

});
}

readURL(count);

See you next time!

MilSats: Medevac

Sometimes, a casualty has to be airlifted to a hospital, stat.

PRT Farah Conducts Medical Evacuation Training with Charlie Co., 2-211th Aviation Regiment at Forward Operating Base Farah
Security force team members for Provincial Reconstruction Team (PRT) Farah wait for a UH-60 Blackhawk medevac helicopter to land before moving a simulated casualty during medical evacuation training on FOB Farah, Jan. 9. PRT Farah coordinated with C Company, “Northstar Dustoff,” 2-211th Aviation Regiment (Air Ambulance) pilots and medics in order to provide invaluable hands-on medical evacuation training. PRT Farah’s mission is to train, advise and assist Afghan government leaders at the municipal, district and provincial levels in Farah province, Afghanistan. Their civil military team is comprised of members of the U.S. Navy, U.S. Army, the U.S. Department of State and the Agency for International Development (USAID). (U.S. Navy photo by HMC Josh Ives/released)

FreeCodeCamp and Learning Node: HTTP Collect

We’ve completed part 1 of the challenges for FreeCodeCamp’s Node section, and now we’re on to Part 2.  FreeCodeCamp broke this up into sections since the Node tutorial is pretty long.  Last time we learned about using http and the get method, and this lesson builds on that.  As always, you should try it first and then look at what I did. Let’s get right to it!

Here are the directions and hints:

HTTP COLLECT (Exercise 8 of 13)
  Write a program that performs an HTTP GET request to a URL provided to you as the first command-line argument. Collect all data from the server (not just the first “data” event) and then write two lines to the console (stdout).
  The first line you write should just be an integer representing the number of characters received from the server. The second line should contain the complete String of characters sent by the server.
 ─────────────────────────────────────────────────────────────────────────────
 ## HINTS
  There are two approaches you can take to this problem:
  1) Collect data across multiple “data” events and append the results together prior to printing the output. Use the “end” event to determine when the stream is finished and you can write the output.
  2) Use a third-party package to abstract the difficulties involved in collecting an entire stream of data. Two different packages provide a
  useful API for solving this problem (there are likely more!): bl (Buffer List) and concat-stream; take your pick!
  To install a Node package, use the Node Package Manager npm. Simply type:
     $ npm install bl
  And it will download and install the latest version of the package into a subdirectory named node_modules. Any package in this subdirectory under your main program file can be loaded with the require syntax without being prefixed by ‘./’:
     var bl = require(‘bl’)
  Node will first look in the core modules and then in the node_modules directory where the package is located.
  If you don’t have an Internet connection, simply make a node_modules directory and copy the entire directory for the package you want to use from inside the learnyounode installation directory:
  file:///home/ubuntu/.nvm/versions/node/v4.4.3/lib/node_modules/learnyounode/node_modules/bl
  file:///home/ubuntu/.nvm/versions/node/v4.4.3/lib/node_modules/learnyounode/node_modules/concat-stream
  Both bl and concat-stream can have a stream piped in to them and they will collect the data for you. Once the stream has ended, a callback will be fired with the data:
     response.pipe(bl(function (err, data) { /* … */ }))
     // or
     response.pipe(concatStream(function (data) { /* … */ }))
  Note that you will probably need to data.toString() to convert from a Buffer.
  Documentation for both of these modules has been installed along with learnyounode on your system and you can read them by pointing your browser here:
  file:///home/ubuntu/.nvm/versions/node/v4.4.3/lib/node_modules/learnyounode/docs/bl.html

  file:///home/ubuntu/.nvm/versions/node/v4.4.3/lib/node_modules/learnyounode/docs/concat-stream.html

Okay, let’s break this down step by step.  I didn’t want to have to use a bunch of appends, because that didn’t sound elegant to me.  I went with one of the third party methods instead, and I picked the Buffer List package.  You can always use the other one.

Type “npm install bl first” at the $ prompt (in other words, NOT in your program), and you’ll get:

<YOUR USERNAME>:~/workspace $ npm install bl
npm WARN package.json chat-example@0.0.0 No repository field.
npm WARN package.json chat-example@0.0.0 No license field.
bl@1.1.2 node_modules/bl
└── readable-stream@2.0.6 (string_decoder@0.10.31, process-nextick-args@1.0.7, util-deprecate@1.0.2, inherits@2.0.1, core-util-is@1.0.2, isarray@1.0.0)

Now Buffer List is installed and you can use it!

Time to start the program.  This is going to use the get method of http again, so we can borrow from the last exercise.  We also need to require the bl package as noted in the hints.  I left require fs in there but you don’t need it for this exercise.

var bl = require('bl');
var fs = require('fs');
var http = require('http');
var addr = process.argv[2];
Remember that the third element of process.arg is the path to the data we want.  That takes care of the variables and requires, so let’s move on to the rest of the program.
http.get(addr, function callback(response) {

&amp;nbsp; &amp;nbsp; &amp;nbsp;/* Do stuff here */

}).on('error', (e) =&amp;gt; {
&amp;nbsp; console.log(`Got error: ${e.message}`);
});

The framework of the program hasn’t changed.  What IS going to change is the part where I put “/* Do stuff here */”. I put the line in the hint in there:

response.pipe(bl(function (err, data) { /* ... */ }))&amp;nbsp;

Just for fun I put

console.log(data.length);

in there, and got a warning that it expected an error to be handled.  From one of the previous lessons, I remembered that we add:

if(err)
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; console.log(err);

I ran it then, even though it’s incomplete, and lo and behold I got a length back that matched the expected answer!  Almost there!  Now, the next thing is to print the contents of the buffer, and they tell you how to do that with a little hint that you need to convert it to a string, like so:

console.log(data.toString());

Do that, and the program passes!

 

Just a quick aside:  It actually took me a half hour and several false trails before I got this.  Believe me, I’m making it seem much easier than it was here!

Anyway, here’s the entire program!

var bl = require('bl');
var fs = require('fs');
var http = require('http');
var addr = process.argv[2];

http.get(addr, function callback(response) {

response.pipe(bl(function (err, data) {
if(err)
console.log(err);

console.log(data.length);
console.log(data.toString());

}))


}).on('error', (e) => {
console.log(`Got error: ${e.message}`);
});