One of the stipulations of the Facebook to Disqus transition was that the Online Team would transfer a year’s worth of Facebook comments over to Disqus so that they would still remain on the website. Facebook doesn’t natively support any kind of data export feature for their commenting platform at the time of writing. I don’t expect that they will in the future. However, they provide a reliable API to their comments database. With a bit of ingenuity and persistence, we were able to successfully import over 99% of existing Facebook comments into Disqus.

Overview of the process

Disqus supports imports in the form of custom WXR files. These are WordPress-compatible XML files that contain things about posts (titles, dates, preview, id, etc.) and comments (name, date, IP, content, etc.).

The Daily Cal uses WordPress and the official Disqus WordPress plugin. The plugin identifies threads with a combination of WordPress’s internal post ID and a short permalink. Thread identifiers look like this:

var disqus_identifier = '528 https://rogerhub.com/~r/code.rogerhub/?p=528';

This one is taken right from the source code of this post (you can see for yourself).

Facebook, on the other hand, identifies threads by the content page’s URL. After all, their system was created for arbitrary content, not just blogs. The Facebook Open Graph API provides a good amount of information about comments. There’s enough information to identify multiple comments posted by a single user. There’s accurate timestamp and reply relationships. There isn’t any personal information like IP addresses, but names are provided.

The overall process looked like this:

1. On dailycal.org, we needed an API endpoint to grab page URLs along with other information about threads on the site.
2. For each of these URLs, we check Facebook’s Open Graph API for comments that were posted on that URL. If there are any, then we put them into our local database.
3. After we are done processing comments for all of the articles ever published on dailycal.org, we can export them to Disqus-compatible WXR and upload them to Disqus.

This seems like a pretty straight-forward data hacking project, but there were a couple of issues that we ran into.

Nitty-gritty details

The primary API endpoint for Facebook’s Comment API is graph.facebook.com/comments/. This takes a couple of GET parameters:

• limit — A maximum number of comments to return
• ids — A comma-delimited list of article URLs
• fields — For getting comment replies

The API supports pagination with cursors (next/prev URLs), but to make things more simple, we just hardcoded a limit parameter of 9999. By default, the endpoint will return only top-level comments. To get comment replies, you need to add a fields=comments parameter to the request.

You can make a few sample requests to get a feel for the structure of the JSON data returned.

Disqus supports a well-documented XML-based import format. In our case, we decided that names were sufficient identification, although Disqus will support some social login tagging in the import format as well. The format specifies a place for the article content, which is an odd request, since article content is usually quite massive. We decided to supply just an excerpt of the article content rather than the entire page.

There were a few more precautions we took before we started development. In order to lower suspicion around our scraping activity on Facebook as well as our own web application firewall (WAF), we grabbed the user agent of a typical Google Chrome client running on Windows 7 x86-64, and used that for all of our requests. We also created a couple of test import “sites” on Disqus, since the final destination of our generated WXR was the Disqus site that we used a year ago before the switch to Facebook. There isn’t any way to copy comments or clone a Disqus site, so we didn’t want to make any mistakes.

Unicode support and escape sequences

The first version of our program had terrible support for any kind of non-ASCII character. It’s not that our commenters were all typing in Arabic or something (actually, we had a couple of comments that really were in Arabic). Smart quotes are used in plain English, and they ruin the process as well.

Facebook’s API spits out JSON data, and uses JSON’s encoding. For example, the double left quotation mark, otherwise known as lrquo in HTML/XML, is encoded as \u201c using JSON’s unicode standard. However, the JSON data also contains HTML-encoded entities like &. (Update: It appears that Facebook has corrected this issue.)

Python’s JSON library will take care of JSON escape sequences as it decodes the string into a native dictionary. However, the script applies HTML entity decoding on that result, in case there are any straggling escape sequences left. Since Disqus’s WXR format suggests that you throw the comment content into a CDATA block, all you need to escape is the CDATA ending sequence, ]]>. You can do this by splitting it up into 2 CDATA sections (e.g. ]]]><![CDATA[]>)

HTTP exceptions

Our API endpoint would timeout or throw an error every once in a while. To make our scraper more robust, we set up handlers for HTTP error responses. The scraper would retry an API request at least 5 times before giving up. If none of the attempts are successful, the URL is logged to the database for further debugging.

Commenter email addresses

Every commenter listed in the WXR needs an email address. Comments with the same email address will be tied together, and if somebody ever registers a Disqus account with that email address, they can claim the comments as their own (and edit/delete them). Facebook provides a unique ID that will associate multiple comments by the same person. But since Facebook comments also allows AOL and Yahoo! logins, not every comment has such an ID. Our script used the Facebook-provided ID when it was present, and generated a random one outside of Facebook’s typical range when it wasn’t. All of the emails ended with @dailycal.org, which meant that we would retain control over the registration verification, in case we needed it.

Edge cases

Disqus requires that comments be at least 2 characters long. There were a couple of Facebook comments that consisted of just one word: “B” or “G” or the like. These had to be filtered out before the XML export process.

We also ran into a case where a visitor commented “B<newline>” on the Facebook comments. For Disqus, this still counts as one character, since the CDATA body is stripped of leading and trailing whitespace before processing. The first version of our script didn’t strip whitespace before checking the length, so it failed to filter out this erroneous comment.

Timezone issues

After a couple of successful trials, we created a test site in Disqus and imported a sample of the generated WXR. Everything looked good until we went and cross-referenced Disqus’s data with the Facebook comments that were displayed on the site. The comment times appeared to be around 5 hours off!

Here in California, we’re GMT -0800, so there wasn’t a clear explanation why the comment times were delayed. WXR specified GMT times, which we verified were correct. The bug seemed to be on Disqus’s end. We contacted Disqus support and posted on their developer’s mailing list, but after around a week, we decided that it would be easiest to just counteract the delay with an offset in the opposite direction.

import datetime
time_offset = datetime.timedelta(0, -5*60*60)
comment_date = (datetime.datetime.strptime(
    comment['created_time'], "%Y-%m-%dT%H:%M:%S+0000"
    ) + time_offset).strftime("%Y-%m-%d %H:%M:%S")

Conclusion

After a dozen successful test runs, we pulled the trigger and unloaded the WXR onto the live Disqus site. The import process finished within 5 minutes, and everything worked without a hitch.

I recently released the WordPress theme that runs this blog on the free public WordPress Theme Repository. The theme repository has very high quality-assurance standards and a team of theme reviewers, who run test suites on each newly submitted theme as well as updates to approved themes. While I was preparing my theme for public release, I picked up on a lot of good CSS tips that helped me improve the maintainability of my theme’s CSS. I hope they will help you too:

0. Use a CSS Preprocessor

Before you continue reading, the first thing you should do is make sure you’re using a CSS preprocessor like SASS. CSS preprocessors are tools that let you write your CSS in a special stylesheet language, and then translate your code into actual CSS before it goes into production. They add programming language features to CSS like variables, nested blocks, mixins (CSS functions), inheritance, and single-line comments.

I choose SASS over other options like LESS or SASS+Compass because:

• SASS is highly stable (it’s built in Ruby) and contains zero bullshit.
• SASS supports an indentation-based syntax similar to python and yaml. LESS does not.
• SASS requires zero configuration, as it should.

It’s easy to set up a loop that re-compiles your SASS when you save changes to it, using the tools I wrote about in my continuous integration post.

1. Stop nesting. Use classes.

When you write CSS, always remember to separate structure and presentation. You should not be adding extraneous elements to your HTML for presentation purposes. This also means that you should disentangle your CSS rules from the structure of your HTML. This is not okay:

<body>
  <header>
    <hgroup>
      <h1>Welcome to my site!</h1>
    </hgroup>

...

// Don't do this!!
body > header
  display: fixed
  top: 0
  hgroup
    h1
      color: white
      font: 700 2.25em/1.5 $serif

As a rule of thumb, you should only nest blocks like this when you mean it. A long selector like body > header div h1 is hardly ever what you want. This CSS will break if you ever move your HTML elements around or add/remove a wrapper anywhere.

Use classes liberally. Your CSS should not look like just an outline of your HTML.

In the long run, classes go a long way towards maintainable CSS. You can always do a recursive search through CSS files to look for class names, but you will have a hard time locating a CSS block like the one above if all you have is the resulting 4-part CSS selector. The above example could be better written like with classes like this:

<body>
  <header class="site-header">
    <hgroup>
      <h1 class="site-name">...</h1>
    </hgroup>
...

.site-header
  display: fixed
  top: 0
.site-title
  color: white
  font: 700 2.25em/1.5 $serif

On that note, you should also stop leaving <div class="clear-fix"></div> around in your HTML. I’ll show you how to work around this with CSS Inheritance:

2. Selector inheritance is awesome

Inheritance is a very powerful, underrated feature in SASS. (LESS doesn’t support it in the way that SASS does.) Selector inheritance allows you to make classes that “extend” other classes, like parent and child classes do in OOP. It doesn’t do this by stupidly copying their CSS rules, but by adding extra selectors to the parent class’s CSS blocks. Let me show you an example:

.content
  margin: 1em 0 1.5em
  p
    color: white
    margin-bottom: 1.5em
.summary
  @extend .content
  margin-bottom: 2em

...

// Would compile to...
.content, .summary {
  margin: 1em 0 1.5em;
}
.content p, .summary p {
  color: white;
  margin-bottom: 1.5em;
}
.summary {
  margin-bottom: 2em;
}

Any rule and child-rule applied to the parent selector will also be applied to the child. Child classes can also have their own properties that override the parent’s. Here’s a more realistic example of what you can do with selector inheritance:

You’ve probably run into the clearing float CSS problem before. If you’re not familiar with it, take a look at this sample code:

<div class="container">
  <div class="left">
    ...
  </div>
  <div class="right">
    ...
  </div>
</div>

...
.container
  border: 1px solid black

.left
  float: left
.right
  float: right

The intended effect is that the container’s 1px black border surrounds both .left and .right, but it appears that only the top border is displayed. The problem arises because floated elements are taken out of flow and now, .container has no height. There are several solutions to this problem, but the most common one involves adding an extra element after both floated div’s and giving it clear: both. Instead of adding extraneous presentation elements to the HTML, you can use selector inheritance:

<div class="container">
  ...
</div>

...
.after-clear-fix
  // "&" refers to the current selector
  &:after
    clear: both
    content: '.'
    display: block
    height: 0
    visibility: hidden

.container
  @extend .after-clear-fix

If you apply @extend .after-clear-fix to several elements, it will compile to a single long CSS selector whose body contains the clearfix rules, thus reducing redundancy in the final stylesheet:

.after-clear-fix:after,
.container:after,
.content:after,
.search-area:after {
  ...
}

SASS also supports a custom syntax that prevents the original class from being printed. Another powerful SASS feature is the @import directive, which works just like CSS’s import directive, but actually brings in the content of the target stylesheet if it is available locally.

3. Organizing your sass/ directory

Most WordPress themes have just 1 stylesheet that’s located at the theme root. When using SASS, my theme stylesheet is usually generated with SASS, while the SASS source files are located in a separate directory. SASS supports the idea of CSS partials. Partials are SASS files whose names begin with an underscore. They are meant to be imported into other stylesheets, and not compiled directly by themselves (although you can do that if you want).

The @import directive is also partial-aware and will match @import "reset" with a file named _reset.sass.

For larger projects, you should split your CSS into files that reflect the semantic role of the parts of the page they style. Additionally, you will usually have a few extra SASS source files for a CSS reset, JavaScript plugins, color/font variables, and mixins. Here’s my theme’s sass directory as an example:

• style.sass — The target of the SASS compiler. WordPress theme metadata is stored in CSS comments, so I put my metadata at the top of this file. This source file also imports the rest of your source files.
• _reset.sass — I use Eric Meyer’s CSS reset. It’s available in SASS form on Google.
• _variables.sass — It’s useful to store all of your colors and fonts in one place and refer to them by variable names. In a monochrome theme like this one, I used color variables named $blue1 to $blue15 in order of increasing lightness.
• _cobalt-global.sass — Buttons, input elements, and WordPress core classes
• _cobalt-grid.sass — Grid for desktop and mobile layout
• _cobalt-header.sass
• _cobalt-primary.sass — The primary content area
• _cobalt-secondary.sass  — Secondary content like sidebars
• _cobalt-footer.sass

You can see that this naming scheme reflects traditional source code a lot more than CSS usually does, and because of the import directive, you don’t sacrifice any performance by splitting your CSS into files like this. One mistake that beginner CSS programmers sometimes make is separating the rules for a single selector into different files. They organize their stylesheets by the semantics of the CSS properties, rather than the semantics of their selectors.

// Don't do this!!
//_colors.sass
.header
  color: black
.content
  color: #CCCCCC
...

//_typography.sass
.header
  font: 700 2.25em/1.5 $serif
.content
  font: 400 1em/1.68 $sans-serif

When you split up your stylesheets like this, you end up having to edit 5 or 6 files even if you’re just working on one part of the site design. Usually there’s also a _mobile.sass thrown into the mix, which is also a bad idea. On the other hand, you don’t want to be writing @media only screen and (max-width: 767px) more than once. You can avoid both of these problems by using SASS mixins:

4. Essential mixins for any project

One of the reasons I don’t like SASS+Compass is that there is too much hand-holding and mixins that you’d never use. Most useful mixins are ones that you can program yourself in a few lines. Here are the essential ones that you might actually find useful:

Mobile Layout Mixin

$mobile1: 767px
$mobile2: 479px

= mobile1
  @media only screen and (max-width: $mobile1)
    @content

= mobile2
  @media only screen and (max-width: $mobile2)
    @content

This is the solution to the mobile layout problem I mentioned in the previous section. Mixins in SASS, as you can see, are defined with the equal sign. The SASS content directive is similar to the Ruby yield directive. Instead of typing out the mobile media query every time you need to use it (or even a lengthy variable interpolating selector), you can use the content directive in SASS 3.2+ to define your responsive mobile styles. Here’s an example of it in action:

.site-navigation
  float: right
  +mobile1
    float: none
  li
    display: inline-block
    +mobile2
      display: block

You should never make a separate SASS source file just for mobile CSS. Instead, mobile styles should be kept with the original non-mobile CSS so that you can see both the original and the overriding properties in a single place.

Vendor Prefix Mixin

= vendor-prefix($name, $argument)
  #{$name}: $argument
  -webkit-#{$name}: $argument
  -moz-#{$name}: $argument
  -ms-#{$name}: $argument
  -o-#{$name}: $argument

A vendor prefix mixin is always useful when when using CSS3 properties like border radius and transitions. However, remember not to depend on CSS3 properties for critical parts of the page. Occasionally, you will need to specify the argument on a separate line if it contains commas:

.button
  $transition: background 0.3s, color 0.1s
  +vendor-prefix(transition, $transition)

That’s it. These two are really the only mixins that you may ever need, and you could have written them yourself in a minute. As a rule of thumb, you should only turn a rule into a mixin if you’re using it at least 3 times, and if a mixin only has 1 property, you might as well use a variable.

5. Prefixes for your classes

Anyone who has worked on a large project has come across an HTML element with a class whose purpose was completely unknown. You want to remove it, but you’re hesitant because it may have some purpose that you’re not aware of. As this happens again and again, over time, your HTML become filled with classes that serve no purpose just because team members are afraid to delete them. The problem is that classes are generally given too many responsibilities in front-end web development. They style HTML elements, they act as JavaScript hooks, they’re added to the HTML for feature detections, they’re used in automated tests, etc.

— engineering.appfolio.com

This quote is from an article about CSS architecture by Philip Walton. He recommends that you prefix classes used by JavaScript hooks with .js- and so on, so that you can change CSS class names without breaking anything. I think it’s a great idea, and I’ve been using these other class prefixes as well: (most of these are for WordPress)

• .site- — For classes that apply site-wide like site headers and footers.
• .entry- — For classes that apply to a single element, of which there are many.
• .nav- — For classes on navigation elements that apply to many entry elements.
• .var- — For variants of classes (e.g. var-blue or var-small)

Intent-based class prefixes like these are advantageous over colloquial class names like .next-button or position-based class prefixes .bottom-navigation.  They let you form a natural class hierarchy using dashes so you can be sure that class names you create are unique. Above all, you don’t want to use generic class names like .container or .header without prefixes. They don’t convey much information about their intent, and they are easy to reuse accidentally, especially if you plan on using messy nested selectors.

6. Know your CSS shorthands

CSS is a vertical language. Lines are rarely more than 75 characters long, but blocks can span 20 or 30 lines easily. To conserve space on your screen (and the screens of those who are reviewing/maintaining your code), use CSS shorthands liberally. Here are a few essential shorthands forms that you should know by heart:

Box Model Shorthand

Numeric box-model properties like margin, padding, and border offer a shorthand notation. The order is clockwise from top, as demonstrated below:

.button
  padding-top: 2px
  padding-right: 3px
  padding-bottom: 4px
  padding-left: 5px

// Equivalent to:
.button
  padding: 2px 3px 4px 5px

You can also specify just 2 or 3 values instead of all 4. In this case, the missing sides will have the same value as their opposite side:

.button
  margin: 4px 3px
  border-width: 1px 2px 0

  // Equvalent to:
  margin-top: 4px
  margin-bottom: 4px
  margin-right: 3px
  margin-left: 3px

  border-width-top: 1px
  border-width-right: 2px
  border-width-left: 2px
  border-width-bottom: 0

Border Shorthand

Borders have 3 properties that are easily distinguishable. They also offer a shorthand form to write all three in one property. Note that you can’t combine the box-model shorthand with this one:

.button
  border-width: 1px
  border-style: solid
  border-color: #00477D

// Equivalent to:
.button
  border: 1px solid #00477D

Font Shorthand

The font property shorthand is the least-commonly used of these shorthands. It’s a hard one to remember, but it will knock out 3 or 4 lines of CSS every time you use it:

$serif: Georgia, Times, serif

.site-header
  font: 2.25em $serif
  font: 2.25em/1.6 $serif
  font: bold 2.25em/1.6 $serif
  font: 700 italic 2.25em/1.6 $serif

  // Final form is equivalent to:
  font-weight: 700
  font-style: italic
  font-size: 2.25em
  line-height: 1.6
  font-family: $serif

You can omit properties from the font shorthand as shown above. However, it must have at least the font size and the font family declarations in order to work at all. If you’re not familiar with numeric font weights, 700 means bold.

Background

The background shorthand is also rarely used because it is hard to remember. Backgrounds have just 4 properties: color, image, repeat, and position. Specify them in that order:

body
  background: blue url(...) repeat-x center center

  // Equivalent to:
  background-color: blue
  background-image: url(...)
  background-repeat: repeat-x
  background-position: center center

That being said, you should not use these shorthands if you only want to specify the top margin or the font size. CSS shorthands are most effective when most of their properties are utilized.

7. Alphabetize for easy comprehension

People have all sorts of different methods they use to order CSS properties. I tend to favor alphabetical order by property name, even if related properties like top and left are separated. You can take a look and pick which one you like yourself:

// Alphabetical order
.search-button
  @extend .button
  background-color: $blue
  border: none
  color: $white
  cursor: pointer
  font: 300 1em/1.25 $sans-serif
  left: 1em
  padding: 4px 9px
  position: relative
  text-align: center
  text-decoration: none
  top: 0.25em
  +vendor-prefix(border-radius, 0.25em)
  vertical-align: middle

Others prefer to order properties by theme:

// Thematic order
.search-button
  @extend .button

  // Box model
  border: none
  padding: 4px 9px
  +vendor-prefix(border-radius, 0.25em)

  // Positioning
  position: relative
  left: 1em
  top: 0.25em

  // Internal styles
  color: $white
  cursor: pointer
  font: 300 1em/1.25 $sans-serif
  text-align: center
  text-decoration: none
  vertical-align: middle

However you decide to order your CSS properties, the effect is the same. All of the tips listed here are purely for the programmer’s convenience. Presentation can always be achieved without CSS preprocessors or shorthands, but using these tools will make your CSS much more readable and maintainable.

Update

I don’t use the plugin mentioned in this post anymore.

On this blog, I use the Crayon syntax highlighter for WordPress to render all the code snippets, since this is a programming blog after all. Crayon is one of the more popular highlighting plugins, as clearly demonstrated by the sad condition of its support forum. It comes with a bunch of color schemes including Ethan Schoonover’s extremely popular “solarized” color scheme and a replica of what Github uses (although the plugin’s version contains a tad bit more purple). A light pastel blue would have fit with this blog’s color scheme quite nicely, but definitely not purple. So, I dove deeper.

Crayon stores its highlighting themes as plain CSS in WordPress’s upload directory. That’s the same place that photos and other media go for your posts. I can think of a couple good reasons why this decision makes more sense than loading the CSS directly from the plugin’s folder:

• WordPress has (by necessity) file permissions to write to the uploads directory, which means that if the plugin ever needs to customize themes, it can do that internally.
• There are hooks for CDN’s and such that mirror WordPress’s upload directory, but may not do the same for plugins and other things.
• On network sites, customized color schemes can be site-specific although the plugin is installed and activated network-wide.

However, there’s also one really bad effect (that likely applies only to me) of this, which I spent a good deal of the afternoon debugging. I run nginx on the backend, and its configuration divides WordPress into two regions: one contains wp-login.php and wp-admin, and the other, everything else. This way, I can restrict WordPress cookies to only the former, which is run over SSL using a self-signed certificate, and avoid having the issue where Google indexes everything twice. (Visitors shouldn’t be reading blogs over https anyway.)

It would be nice if everything could be so cleanly divided this way, but wp-admin also uses file uploads like image thumbnails, which is why there are all the mixed-content warnings you may have seen if you’ve ever administered anything through a web panel over https. Browsers complain when you try to load pages with mixed content, but they will let you do so anyway. Things are different, however, when the request is made after the page has loaded, through AJAX.

There’s an inherent security problem with letting a page script load whatever resources it wants to from anywhere on the Internet. This issue is addressed by the same-origin policy framework, wherein the server with the resource (as opposed to the one containing the web page) gets to decide if access is allowed.

HTTP/1.1 200 OK
Access-Control-Allow-Origin: http://example.com

The header takes either a list of origins, null, or the wildcard *. It’s ultimately up to the browser to implement this security header, but most modern ones do.

Now, here’s the twist: Crayon uses the built-in WordPress function, wp_upload_dir(), to find and load custom themes. If your server is set up like mine, this will return a http scheme address, whereas a https address is required. Without a check to see if the page is loaded through SSL, the plain-text AJAX call will fail because browsers treat the SSL and the plain-text version of a site as different origins.

There are a couple ways to go about this. You can:

• Disable SSL temporarily while you’re editing the custom color themes (probably the easiest).
• Instruct the browser to ignore same-origin policy.
• Patch the plugin code to use https where appropriate.

I ended up going with the third, after unsuccessfully attempting to send a access-control-allow-origin header with every response on the server-side. This was a rather frustrating issue to resolve, but it was interesting to see the kinds of problems that arise in exchange for progress in security.