Email Basics: The Internet Message Format

Leave a reply

Note: This is part 2 of a two-part segment on Email Basics. You can also read part 1 of this segment, which deals with the Simple Mail Transfer Protocol.

In part 1 of this two-part segment, I mentioned that the goal of SMTP was to ensure the responsible delivery of a digital package, but also that SMTP does not care what kind of data the package contains. In fact, a good implementation of SMTP should accept any sequence of bits as a valid package so long as it ends in <CRLF>.<CRLF>, which is 0D 0A 2E 0D 0A in hexadecimal. In practice, this package of data usually follows the RFC822 standard or its successors, which define the format of “Internet Text Messages”, more commonly thought of as Email.

RFC822 and SMTP were designed to be very robust protocols. They were designed to:

keep memory usage contained in the case of Email servers that may run for thousands of days
be human-readable for debugging purposes
support backward-compatibility as new features are added

These properties will become clear as we talk about RFC822. Before I start, you should know that you can view the full, original RFC822 version of any email you have ever received. These source messages are available in Gmail with the “Show Original” dropdown menu option and in Yahoo Mail with the “Show Headers” option. If you open up one of your emails, you will notice a few things:

The actual email is at the bottom. The top has a bunch of Key: Value pairs, some of which you recognize and all of which are in English.
At the bottom, your email is repeated twice. One version looks like a normal text email, but the other one has a bunch of HTML-like tags embedded in it.
(If there are any attachments) There is a huge chunk of gibberish at the bottom.

(If you’ve ever worked with HTTP headers, then Email headers should look very familiar to you. They work mostly the same way, with some small differences.)

Email headers are bits of information that describe an email. They appear at the top of an email in Key: Value pairs. They can span more than 1 line by prefixing the continuing lines with one or more whitespace characters. Email headers work on the Collapsing White Space (CWS) idea, like HTML does. Any number of spaces, tabs, CRLF’s, or sequences of whitespace are treated as a single space character. Email headers include the familiar From:, To:, Subject:, and Date: headers, but they also include these important pieces of information: (try to find these in your own emails!)

Message-ID – Uniquely identifies the email so that we can have things like threads using emails. They look like email addresses, but have a more restrictive syntax.
References or In-Reply-To – List of Message-IDs that this email is related to. Used for making threads.
Received – This header is added on to the beginning of the email headers whenever the email changes hands (by means of SMTP or other means). It contains information about the receiving server and the sending server, as well as the time and protocol variables. The newest ones are always at the top, not the bottom.
DKIM-Signature – A PKI signature by the Email’s sender’s mail server that guarantees the email has not been tampered with since it left its origin. More on this later.
X-* – Any header that begins with X- is a non-standard header. However, non-standard headers may serve some very important functions. Sometimes, standards begin as X- non-standard headers and slowly make their way into global adoption.

After the email headers is a completely empty line. This marks the border between the headers and the email body. Most emails today are transferred using the multipart/alternative content-type. The body of an email is sent in 2 forms: one is a plain text version that can be read on command-line email clients like mutt, and the other is a HTML version that provides greater functionality in graphical email clients like a web application or desktop email application. It is up to the email program to decide which one to show.

Applications are typically encoded in Base64 before they can be attached to an email. Base64 contains 64 different characters to represent data: lowercase and uppercase letters, numbers, and the + and / symbols. Attachments just follow as another block in the multipart/alternative encoding of email bodies. (Because of this, attachments are actually 4 times as large as their originals when they are sent with an email.)

Here’s an example RFC822 email that demonstrates everything up to now:

Received: by castle from treehouse for
    <lemongrab@castle>; Tue, 25 Jun ...
Message-ID: <haha+man@treehouse>
Date: Tue, 25 Jun 2013 11:29:00 +0000
Subject: Hey!
Return-Path: Neptr <neptr@mybasement>
From: Finn the Human <fth@treehouse>
To: Lemongrab <lemongrab@castle>
Content-Type: multipart/alternative; boundary=front

--front
Content-Type: text/plain; charset=UTF-8

mhm.. mhmm...
You smell like dog buns.

--front
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<p>mhm.. mhmm...<br />
You smell like dog buns.</p>

--front--

When your mail server receives an email, it has no way of knowing if the sender is really who he or she claims to be. All the server knows is who is currently delivering the email. To help discourage people from forging these email headers, two infrastructural standards were developed: SPF and DKIM.

Both of these standards depend on special pieces of information made available in a domain’s DNS records. SPF (Sender Protection Framework) lets a domain define which IP addresses are permitted to deliver email on its behalf. (This works because, in reality, a TCP connection is usually established directly between the origin and the destination servers without any intermediate servers.) Email from a server that isn’t in a domain’s SPF record is considered suspicious. Gmail displays the result of a successful SPF verification by saying an email was Mailed by: ... a domain.

To see a server’s SPF record, perform the following:

$ nslookup -q=txt tumblr.com
> ...

DKIM is public-key cryptography applied to email. A server releases one or more public keys through its DNS records. Each public key is available through a TXT DNS lookup at key._domainkey.example.com.

$ nslookup -q=txt 20120113._domainkey.gmail.com
> 20120113._domainkey.gmail.com	text = "k=rsa\; p=MIIB
> ...

The origin server declares which headers are “signed headers” and then applies its signature to both the signed headers and the email body. Gmail displays the result of a successful DKIM verification by saying an email was Signed by: ... a domain.

It is important to note that any Internet-connected computer can act as a origin server for email. However, only servers designated by a DNS MX record can be configured to act as the receiving server. When you are designing your own email-enabled web applications, it is important to keep these authentication mechanisms in mind to ensure that your email comes only from trustworthy SMTP origin servers and that you take proper actions to prevent your emails from being classified as spam.

Email Basics: Introduction to SMTP

The SMTP Conversation

Package delivery through SMTP works like a conversation. This conversation takes place between a client, who has a package to deliver, and a server, who is receiving the package. Once the package is received by the server, the client can forget about it and the server assumes full responsibility for the package and its delivery. In this way, servers sometimes become clients themselves if they need to forward packages on to other servers.

Like any good conversation, SMTP begins with an introduction by both parties:

Server: 220 localhost Greetings
   You: HELO roger
Server: localhost

In case you’ve got bad memory, the server gives you its name twice. Once when you open the connection, and again after you introduce yourself with a HELO. The names used here are special names known as hostnames. In real-world SMTP, these hostnames are usually fully-qualified domain names (FQDN’s) and look like mail.rogerhub.com.

You’ve both said helo. Now we get down to business. The usual thing to do at this point is to tell the server about the package you’re delivering. Of course, if you’re shy you can just walk away.

   You: QUIT
Server: 221 Service closing transmission channel

To declare that you’ve got a package to deliver, the MAIL FROM command is used. The command is followed with a single colon and then the email address of the sender.

   You: MAIL FROM:Finn the Human <finn@treehouse>
Server: 250 OK

Next, you declare the recipients of the package with RCPT TO. If there are more than one, which happens often, then you declare each recipient one at a time.

   You: RCPT TO:Jake the Dog <jake@treehouse>
Server: 250 OK
   You: RCPT TO:BMO <bmo@treehouse>
Server: 250 OK

So far so good. We are not far from the end. Once the formalities are exchanged, you can begin delivering the package with the DATA command. After you’ve delivered the whole package, you tell the server that you’re done by sending a single period on its own line.

   You: DATA
Server: 354 Start mail input; end with <CRLF>.<CRLF>
   You: Jake,

        Left the house for a few hours. Will be back
        soon.

        Not joking,
        Finn
        .
Server: 250 OK

That’s it! At this point, you can say goodbye with QUIT or send another email. (These command responses are straight from a demo SMTP server implementation on my GitHub.)

But this can’t be all there is! Is there more to SMTP? You bet there is!

In fact, SMTP and RFC822 have both seen many revisions to the specification over the years. One of the major changes to the specification is in the initial greeting. Instead of HELO, an alternate greeting EHLO, or extended helo, was proposed. When you greet an EHLO-aware server with EHLO, it will send you an extended SMTP (or ESMTP) response like so:

Server: 220 localhost Greetings
   You: EHLO roger
Server: 250-localhost
Server: 250-PIPELINING
Server: 250 SIZE 10000000

The server sends a list of SMTP extensions that it supports. These include things like STARTTLS, which provides a way to upgrade a plaintext connection (like all the transmissions seen here) to an anonymous, encrypted TLS session, which is more resilient to snooping. Both the client and the server have to support an extension before you can use it. Extensions open up the path to a some extra-cool stuff like authentication and UTF-8 support.

You can try out this whole process by sending an email to a friend, just like a real mail server would do. The first step is to find the IP address of the server to contact! The hostname of your request is the hostname of the email address (e.g. gmail.com), and the query type is mx, which stands for Mail Exchanger.

$ nslookup -q=mx gmail.com
...

After that, you just need to open a TCP connection to one of the listed Mail Exchangers. You can do this with either netcat or telnet, both of whose availability depend on your operating system.

$ nc example.com 25       # netcat
$ telnet example.com 25   # telnet

Fire away! (But don’t be surprised if your phony emails are discarded as spam.. Haha)

Continuous integration in web development

CLI JavaScript Minification

There are plenty of websites that will do JS-minification for you online. In fact, most of them run in JavaScript themselves. However, if you find yourself visiting them more than once a day, you should get yourself a command-line JavaScript minifier. Dean Edwards, the author of /packer/, one of the most popular JS minification websites online, has ports of his JS packer available in several programming languages on his website. (I’m using the PHP version, because it appears to be the most faithful port.)

After you’ve acquired a minifier and have appended it to your $PATH, you can incorporate it into your build scripts like so:

#!/bin/bash
packer src/final.english.js out.js
./build.py > out.htm
...

inotify Tools

Even if you’ve combined all of your build tasks into a single build script, you still have to run the thing every time you want to see new changes in your web application. A sleep-build loop would take care of this inconvenience, but then you’re stuck between wasting CPU cycles and having to wait a bit for new changes to appear. We can do better.

The term inotify stands for index-node (inode) notification. Most unix systems come with tools that bridge inotify services with the CLI, and you can use these to have your build script run only when you change and save a source file in your text editor.

I have an alias set up like so:

alias watchdir='inotifywait -r -e close_write,moved_to,create'

The switches enable recursive behavior and restrict the events to a certain few. This command will block until one of the three events occurs:

close_write – when a file handle in w mode is closed
moved_to – when a file is moved into the directory
create – when a new file is created in the directory

Combined with an rsync alias with some reasonable defaults, you can put together a loop that syncs source files to a remote server and builds them as they change.

alias rrsync='rsync -vauzh --progress --delete'

For example, I used something like the following while developing this blog’s theme:

while watchdir cobalt; do sass ... ; rrsync cobalt ... ; done

This last one isn’t a concept from CI, but it can be adapted to fit in your CI workflow whenever you need it.

Ad-hoc Web Server

You can test static html files just by opening them locally with your web browser, but there are a few reasons that an actual web server, no matter how simple it may be, is a slightly better option (root favicon, greater network permissions, absolute paths, protocol-relative URL’s, just off the top of my head). Python (and Python 3) comes with a built-in single-threaded simple web server that supports directory listing, symbolic links, a decent number of MIME types, modification/expiration headers, and large-file streaming. In other words, it’s a pretty good tool that will do anything you could want related to serving static assets (you know, unless you want to download two things at once).

Start the python web server at the command line, and it will start serving files through HTTP from the current working directory:

> python -m SimpleHTTPServer [port number]  # Python 2
> python3 -m http.server [port number]      # Python 3

Formatting source code: What universities don’t teach you

2 Replies

Some programmers write code everyday completely unaware of their number-one most essential tool even works: the text editor. There’s a good reason for this. Most compilers and interpreters don’t care how code is formatted, and unless the language you’re working with cares about file encoding or white space, there’s no reason that you’ll ever have cause to investigate further. Here are 3 things every programmer should know about formatting source code:

Carriage Returns, Line Feeds

Have you ever experienced one of these issues?

You open somebody’s source code only to find that everything is on one line.
At the end of every line of code, you see a ^M character.

Depending on your operating system, you’ve probably encountered at least one of these. The problem arises from a disagreement among programmers about how to best represent the beginning of a new line of text. You see, Macintosh computers used to use the Carriage Return character (Char 13, 0x0D) commonly represented by \r to indicate that two pieces of text were on separate lines (carriage refers to part on a typewriter). This conflicted with the Unix convention that the New Line character (Char 10, 0x0A) represented by \n should be used.

To maintain compatibility with both systems, Microsoft Windows decided that both characters should be used wherever a new line of text began, and by convention, the order of the characters was to always be \r\n, and never the reverse.

Newer versions of the Macintosh operating system are Unix-like and no longer use the \r character, but Windows still maintains their double-character convention. To see the difference this makes, compare the following two binary sequences. This one uses the Unix convention (Linux, OS X, etc.):

00000000  59 6f 75 20 67 75 79 73  20 61 72 65 20 73 6f 20
00000010  67 72 65 61 74 2e 0a 48  61 68 61 0a

And this one uses the Windows convention:

00000000  59 6f 75 20 67 75 79 73  20 61 72 65 20 73 6f 20
00000010  67 72 65 61 74 2e 0d 0a  48 61 68 61 0d 0a

The ^M character that you may have seen comes from the fact that Carriage Return’s big-endian binary representation is 000 0101 (0x0D), while Capital M’s representation is 100 0101 (0x4D), different only by their MSB!

Since the Carriage Return convention is no longer popular and the CRLF (Windows) convention is only used by one major, modern operating system, you should stick to the Unix convention when formatting your source code unless you’re implementing a Internet standard that specifies CRLF. On vim, you can check your line-ending convention with :set fileformat or change it with :set fileformat=unix.

No doubt you’ve seen this in a version control commit log:

\ No newline at end of file

It is convention that there is a new line after the last line of a text file. This convention isn’t followed by some text editors. Some will even add a new line upon opening a file, before you even make any changes. In vim, you can toggle this behavior with :set eol binary or :set noeol nobinary.

Tabs vs Spaces

Hitting the Tab button usually inserts one horizontal tab character (\t, Char 9, 0x09), but this isn’t always the case. Sometimes your text editor will insert 4 spaces, 2 spaces, or 8 spaces. Furthermore, your text editor might display a tab character as 2, 4, 8, or any number of characters. While liberal application of tabs and spaces can improve the readability of your code, it is all for naught if your reader’s spaces-per-tab ratio is different from yours.

When you edit existing source code, you should make sure to follow the spacing convention of the existing code. If tabs are used, then use tabs. If spaces are used, use spaces. If they are mixed, complain to the author.

On Vim, there is a plugin called vim sleuth that will detect the spacing style of existing source code and try to match it. It sets several important Vim settings:

:set tabstop (ts) – Your spaces-per-tab ratio.
:set expandtab (et) – Whether or not spaces should be used instead of tabs.
:set shiftwidth (sw) – How many characters your cursor will shift to the right when you hit the Tab key. If shiftwidth is not a multiple of tabstop, then spaces will be used to satisfy the remainder.

Occasionally, it may be useful to adjust these yourself in order to more easily read code that has been formatted with both tabs and spaces (ಠ_ಠ). For your own source code, you should always exclusively use either spaces or tabs, and never both.

File Encoding

If this hasn’t happened to you before, you’ve probably at least seen it around the Internet: apostrophes like the one in Roger’s get turned into funky stuff like Rogerâ€™s. File encodings are exactly why this happens.

The apostrophe used in the example above isn’t Apostraphe, ASCII Char 0x27, but Unicode Character RIGHT SINGLE QUOTATION MARK 0x2019. Unicode is an extended character encoding that provides support for things like Chinese characters and happy faces. It uses different sequences of bits to express different characters.

The only reason why Roger is still readable in the above example is the fact that ASCII characters originally only used 7 bits per character. Obviously, representing thousands of Chinese characters in only 7 bits is impossible, and switching to 16 bits per character would double the size of any text file. This is why other file encodings are variable-length encodings and use the unused 0x80 to 0xFF range of ASCII to indicate control characters, like characters that indicate how many bytes a certain character takes up. The important part is, plain ASCII is always valid in the other encodings.

The issue arises when extended character set encodings are mixed and the programmer forgets to specify which is being used. So, back to the example above. That special quotation mark is a valid unicode UTF-8 character. Here is its hexdump:

00000000  e2 80 99

In binary, it’s:

11100010 10000000 10011001

UTF-8’s run-length encoding works by indicating the length of the character using the position of the first 0 in the character’s binary encoding. Every subsequent data-character begins with 0b10. In this case, the first 0 follows 3 1s, which indicates that this is a 16-bit character composed of 3 bytes. If you add up the remainder bits (4 in the first byte, 6 in the second byte, and 6 in the third byte), you see that they add up to 16 data bits:

11100010 10000000 10011001 (Control bits)

11100010 10000000 10011001 (Data bits)

Reassemble those data bits and you get 00100000 00011001 or 0x2019, the hexadecimal character code I mentioned before!

Many text editors now interpret source code as UTF-8 by default, and only try the ISO-8859-1 standard when it encounters a illegal UTF-8 character. However, web browsers usually do the opposite. Depending on the HTTP Content-Type header and doctype, a web browser may sometimes interpret a document as ISO-8859-1 by default when there is no explicit Content-Encoding specified (though you should always specify this, either in your headers, or better yet, in the HTML itself as a meta tag). The same bits above, when interpreted as an ISO-8599-1 sequence produce:

â€™

ISO-8599-1 is also known as latin-1. Each character is 1 byte long, and there are no multi-byte characters. The ISO-8599-1 encoding of those three characters is e2 80 99, exactly the same as the Unicode encoding from before.

Tumblr’s Phishing Protection Code

1 Reply

At the top of every Tumblr user’s blog, there’s a piece of JavaScript inserted by Tumblr itself. In general, Tumblr is very generous about the control they give you over your blog’s appearance. They don’t insert any advertisements or enforce any global content other than a Quantcast visitor analytics script, follow/dashboard controls on the upper right, and this script. I’ve posted the first few lines here:

(function(){var a=translated_warning_string;var b=function(d){d=d||window.event;var c=d.target||d.srcElement;if(c.type=="password"){if(confirm(a)){b=function(){}}else{c.value="";return false}}};setInterval(function(){if(typeof document.addEventListener!="undefined"){document.addEventListener("keypress",b,false)}},0)})();

This isn’t particularly clever or difficult to understand, but it does utilize several good ideas in JavaScript programming, so I’d like to go over it. First of all, you’ll notice that this extract is a single line of code that’s surrounded by (function() { ... })();. Javascript treats functions as first-class citizens. They can be declared, reassigned, and called inline. There are several advantages of first-class functions in any dynamically-typed language:

The ability to create anonymous functions is particularly helpful when you don’t want to clog up your global namespace with function names that are only used in one part of your code.
They can be created dynamically using, what’s known as a closure in many languages. Closures take their variables from the environment frame in which they were created, so you can generate them on-the-fly in a loop, in an event handler, or as part of a callback.
Even if you don’t need or want any of the fancy benefits listed so far, it’s useful that local variables declared in first-class functions are destroyed when the function quits. That way, you can use names like a or _ without polluting your global namespace.

I’ve reformatted the statements inside the function here:

var a = translated_warning_string;
var b = function(d){
  d = d||window.event;
  var c = d.target||d.srcElement;
  if (c.type == "password"){
    if(confirm(a)){
      b = function(){}
    } else {
      c.value = "";
      return false
    }
  }
};
setInterval(function(){
  if (typeof document.addEventListener != "undefined"){
    document.addEventListener("keypress",b,false)
  }
}, 0)

The first line references the global translated_warning_string variable that is declared in the blog’s HTML itself. Its contents should hint at the goal of this code. (Although it looks like not all languages are supported yet.) Assigning the variable to a means that it can be reused without making the code too long, but it seems that the variable is only actually referenced once here.

Variable b gets assigned to a function in the same way we manipulated functions earlier. The d = d||window.event; is a metaphor for if d is garbage use window.event instead, or else, leave it alone. It is a handy way to specify a default value in variable assignment when you’re uncertain whether the preferred value will work or not. In this case, d defaults to window.event which is a hack for older versions of Internet Explorer. (more on this later)

I was kidding when I said later. Here’s more on it now. Skip ahead a few lines and peek over at this part of the code:

if (typeof document.addEventListener != "undefined"){
  document.addEventListener("keypress",b,false)
}

The function we declared earlier, b, is used as the event listener for the document’s keypress event. It is triggered whenever someone presses a key while on the webpage (actually a little more complicated than this). Event listeners are functions that are supposed to be called with a single parameter, the event’s event parameter, which contains information about the event that triggered the call. This is not the case in older versions of Internet Explorer, hence the d = d||window.event; statement earlier.

When a keypress happens, the event actually travels down the tree to the target element first, from the root to the leaf (document → html → body → …). It gives each of these elements a chance to “capture” the event and handle it before it reaches its destination. This is known as event capturing, and is specified by the third argument to EventTarget.addEventListener. The other more commonly used and default behavior is to let the event bubble up if the destination doesn’t handle it. The event will bubble up until one of the destination’s ancestors catches it and stops the bubbling.

The code examined here chooses not to capture the event on the way down. (If it did, then all events would have to go through this handler.) The trade-off to not interfering with events that already have keypress handlers is that the behavior can be easily overridden by rogue sites (although this is easily detectable).

Back to where we were, we now see that variable d is an event, so c is its EventTarget, and also a dom-tree node. The variable-assignment-with-default trick is used again here.

if (c.type == "password"){
  if(confirm(a)){
    b = function(){}
  } else {
    c.value = "";
    return false
  }
}

If the event’s source is a password input, raise a confirmation dialog with the text stored in a, the translated confirmation string. If the confirmation passes, then b is set to an empty anonymous function, function(){}. Recall that b was previously defined to be another anonymous function and was also used in our event listener. Clearing this variable after the first successful confirmation prevents the script from prompting the user more than once, which is a good idea.

If the user rejects the confirmation, then the password input is cleared and the keyevent is halted with return false. Note that JavaScript dictates that the last semicolon in a block is optional, although it is usually good style to include it anyway.

Finally, note that the event listener is wrapped inside a setInterval(function() { ... }, 0). setInterval is a built-in Javascript function that runs a piece of code regularly. The second parameter is the delay in between calls, specified in milliseconds. In this case, 0 milliseconds is used, but the web browser imposes a lower limit on this delay.

The function’s contents checks the type of document.addEventListener. This function is part of the DOM that is set up near the beginning of every page load. Once the event infrastructure is available, the listener is attached. A more common way to achieve the same affect is to attach an event listener to the window object’s onload function, which is usually achieved through one of many Javascript libraries, although this solution is appropriate in the context of this script.