Fork me on GitHub

User Guide

Sunny provides a multi-cloud common programming interface. As the cloud providers that Sunny abstracts have all chosen different names for equivalent structures, we collapse the terms for cloud folder-like and file-like objects to:

This guide is for Sunny version: v0.0.6.

Configuration

Sunny needs cloud provider secrets, either via a configuration file or the process environment. The goal is to create a sunny.Configuration object, which has a connection member for starting cloud operations.

For more information, see the Configuration API documents.

Secrets

The first thing you need is your cloud data store account and secret keys. Here are some good starting points if you don’t already have these.

Configuration Object / File

Sunny can be set up from a straight JavaScript object. However, for security, it is recommended that you have a separate “secrets” file that is not under version control. Here’s a basic scenario.

First, create a separate file “my-config.js” that will be require‘ed by node in the following format:

module.exports.MyConfiguration = {
  provider: ("aws"|"google"),
  account: "ACCOUNT_NAME",
  secretKey: "ACCOUNT_SECRET_KEY",
  authUrl: "s3.amazonaws.com", // or other optional endpoint
  ssl: (true|false)
};

Then include the file in your application code like:

var myConfig = require("./my-config.js").MyConfiguration,
  sunny = require("sunny"),
  sunnyCfg = sunny.Configuration.fromObj(myConfig);

See Configuration.fromObj for more.

Process Environment

Sunny can be alternatively configured from environment variables:

$ export SUNNY_PROVIDER=("aws"|"google")
$ export SUNNY_ACCOUNT="ACCOUNT_NAME"
$ export SUNNY_SECRET_KEY="ACCOUNT_SECRET_KEY"
$ export SUNNY_AUTH_URL="s3.amazonaws.com"  # or other optional endpoint
$ export SUNNY_SSL=("true"|"false")

Note: It’s probably best to wrap the environment variable export into a script.

Then hook the variables in your application code like:

var sunny = require("sunny"),
  sunnyCfg = sunny.Configuration.fromEnv();

See Configuration.fromEnv for more.

An Example - PUT’ing a File

We’ll start with a known container that already exists – let’s call it “sunnyjs”. We will take our configuration object (sunnyCfg), then:

  1. Get a container object through an asynchronous getContainer().
  2. Write our file through an asynchronous putBlobFromFile().
  3. Get the blob and check the data.

Let’s start with our file:

$ echo -n "Hello Sunny.js..." > my-file.txt

Now let’s do our cloud operations. For purposes of this example, we’ll use the (awesome) Async.js library to serialize certain operations so we don’t get into a ridiculous level of callback nesting. The library is not actually required for Sunny, as the only current install dependency is xml2js. The takeaway point here is the async.series executes each function sequentially, waiting until one is done before moving to the next. Just what we want here.

We’re assuming you have a sunnyCfg object from the previous section.

/* 'sunnyCfg' already assumed to exist. */
var async = require('async'),
  contName = "sunnyjs",
  filePath = "./my-file.txt",
  blobName = "my-blob.txt",
  buf = [],
  connection = sunnyCfg.connection,
  containerObj,
  blobObj,
  request,
  stream;

errHandle = function (err) {
  console.error("Got error: %s", err);
};

async.series([
  function (callback) { // GET container.
    request = connection.getContainer("sunnyjs");
    request.on('error', errHandle);
    request.on('end', function (results, meta) {
      // Set container object, signal async done.
      containerObj = results.container;
      console.log("GET container: %s", containerObj.name);
      callback(null); // Moves async.js to next function.
    });
    request.end();
  },
  function (callback) { // PUT blob from file.
    request = containerObj.putBlobFromFile(
      blobName, filePath, { encoding: "utf8" });
    request.on('error', errHandle);
    request.on('end', function (results, meta) {
      blobObj = results.blob;
      console.log("PUT blob: %s", blobObj.name);
      callback(null); // Moves async.js to next function.
    });
    request.end();
  },
  function (callback) { // GET blob with data.
    stream = blobObj.get({ encoding: 'utf8' });
    stream.on('error', errHandle);
    stream.on('data', function (chunk) {
      // Accumulate data in a data.
      // We get string data because we set an encoding.
      buf.push(chunk);
    });
    stream.on('end', function (results, meta) {
      blobObj = results.blob;
      console.log("GET blob: %s", blobObj.name);
      callback(null); // Moves async.js to completion.
    });
    stream.end();
  }
], function (err) {
  // Finished all async series.
  console.log("Final GET data: \"%s\"", buf.join(''));
});

which produces the following output:

GET container: sunnyjs
PUT blob: my-blob.txt
GET blob: my-blob.txt
Final GET data: "Hello Sunny.js"

completing our round trip from local file to cloud, back to a string.

Cloud Operations

Sunny’s cloud API is based around event emission and listeners. All operations that actually go out on the network will return either a request or stream object that the caller then adds listeners too.

As a side note, a great place to see a lot of examples of all the Sunny cloud operations, is in the source code ”Live Tests”, which perform test operations against real cloud datastores (links to source files are available in the descriptions for the blob, connection, and container test fields).

Finally, its worth pointing out that these example only skim the surface of options, parameters and results that are part of the API. Read the API documentation for the complete descriptions.

Everything from this point on assumes that you have a sunnyCfg object available from the configuration section above.

Connection

The Sunny configuration object has a connection member that can be used to get a list of containers or a single container by name.

Connection.getContainers()

Let’s get a list of all containers and print the first five names to console:

// List all containers in account.
var request = sunnyCfg.connection.getContainers();

// Set our error handler.
request.on('error', function (err) {
  console.warn("ERROR: %s", err);
});

// Set our completion event.
// (The 'results' object varies for operations).
request.on('end', function (results) {
  var containerObjs = results.containers;
  console.log("First 5 containers are:")
  containerObjs.slice(0, 5).forEach(function (container) {
    console.log(" * %s", container.name);
  });
});

// Ending the request actually starts the cloud request.
// If your code seems to be hanging, check that you 'end'-ed
// the request!
request.end();

which produces:

First 5 containers are:
 * bar
 * baz
 * foo
 * sunny
 * zed

(or something of the like).

Connection.putContainer()

Now, let’s create a new container. Note that for both Amazon S3 and Google Storage, there is a global bucket namespace, so you will have to choose a unique name, and might get err object such that err.isNotOwner() === true.

Here’s an example assuming the bucket name doesn’t exist. Notice that the on() events are chainable (although end() currently is not).

sunnyCfg.connection.putContainer('sunnyjs')
  .on('error', function (err) {
      console.warn("ERROR: %s", err);
    })
  .on('end', function (results) {
      var containerObj = results.container;
      console.log("Created container: '%s'.", containerObj.name);
    })
  .end();

If it succeeds, we get:

Created container: 'sunnyjs'.

If you have a container object, the method is also available as Container.put().

Connection.getContainer()

Last, let’s get a single (known) container object from the datastore. Note that by default the option validate is false which means we don’t actually perform a cloud operation (since it isn’t needed). If the container doesn’t really exist, a “not found” error will be thrown on a later blob operation that actually performs a network operation.

To force the error early, setting the options parameter validate to true, we will get an error if the container is not found (err.isNotFound() === true) or something else is invalid with the cloud request or name.

sunnyCfg.connection.getContainer('sunnyjs', { validate: true })
  .on('error', function (err) {
      console.warn("ERROR: %s", err);
    })
  .on('end', function (results) {
      var containerObj = results.container;
      console.log("Found container: '%s'.", containerObj.name);
    })
  .end();

which gives us:

Found container: 'sunnyjs'.

If you have a container object, the method is also available as Container.get().

Container

Once we have a container object (here we’ll assume we’re up to the last callback above and have the containerObj object available), we can manipulate the container (perform DELETE or PUT cloud operation corresponding to the container object’s information), and list corresponding blobs.

We’ve already seen the get() and put() operations on a cloud object from the connection object aliased methods. Let’s look at some other calls for with a container object.

Container.del()

Delete a container:

containerObj.del()
  .on('error', function (err) {
      console.warn("ERROR: %s", err);
    })
  .on('end', function (results) {
      var containerObj = results.container;
      console.log("Deleted container: '%s'.", containerObj.name);
    })
  .end();

which gives us:

Deleted container: 'sunnyjs2'.

Note: If the container did not already exist, there is no error returned, as not all cloud providers through a “not found” error. Instead, the results callback object has a deleted member that if true means that (1) the container was deleted, and (2) we could actually tell if it was deleted (which isn’t always the case).

Container.getBlobs()

The getBlobs operation gets a list of blob objects in a container, filtered by several input parameters (see the API). The call does not retrieve any actual blob data. Let’s say we have the following blobs in a container:

We can use the delimiter parameter to cause the cloud service to treat the flat blob namespace as if it where ”/” delimited, set a prefix of “foo/” which is as if we were within the “foo/” implied directory, and specify a marker of “foo/blob001.txt”, which means our first result should be lexicographically after that result.

containerObj.getBlobs({
      prefix: "foo/",
      delimiter: "/",
      marker: "foo/blob001.txt"
    })
  .on('error', function (err) {
      console.warn("ERROR: %s", err);
    })
  .on('end', function (results) {
    console.log("Blobs found:");
    results.blobs.forEach(function (blob) {
      console.log(" * %s", blob.name);
    });
    console.log("Psuedo-directories found");
    results.dirNames.forEach(function (dirName) {
      console.log(" * %s", dirName);
    })
  .end();

The results callback objects includes a blobs property which is an array of blob objects, as well as dirNames, an array of strings constituting psuedo-directories at the current “level” (here, within the “foo/” implied directory).

Blobs found:
 * foo/blob002.txt
 * foo/blob003.txt
Psuedo-directories found
 * foo/zed/

Container.putBlob()

PUT’ing a blob allows us to create a new blob with data, as well as insert cloud metadata. Instead of a “request” object, the putBlob() method gives us a “stream” object instead, which is an implementation of a Node.js Writable Stream. To add data, we can call write() with strings or Buffer objects, and then call end() to start the cloud request. (Note: end() also optionally takes a string or buffer).

Let’s create a blob named “foo.txt” with some string data and a couple of metadata key / value pairs.

var stream = self.container.putBlob("foo.txt", {
  metadata: {
    'foo': "My foo metadata.",
    'bar': 42
  }
});

// Set handlers.
stream.on('error', function (err) {
  console.warn("ERROR: %s", err);
});
stream.on('end', function (results, meta) {
  var blobObj = results.blob;
  console.log("Wrote data for: '%s'.", blobObj.name);
});

// Write, then cause request to start with 'end()'.
stream.write("Hello there!");
stream.end();

which produces:

Wrote data for: 'foo.txt'

If you have a blob object, the method is also available as Blob.put().

Container.getBlob()

Now let’s get and print both the data and the metadata. In parallel to PUT, GET blob implements the Readable Stream interface.

Note that per the stream interface, because we set an encoding here, the “data” event passes strings instead of the default Buffer objects, which we accumulate into an array, then join and print on the “end” event.

var buf = [],
  stream = self.container.getBlob("foo.txt", {
    encoding: 'utf8'
  });

stream.on('error', function (err) {
  console.warn("ERROR: %s", err);
});
stream.on('data', function (chunk) {
  buf.push(chunk);
});
stream.on('end', function (results, meta) {
  var blobObj = results.blob;
  console.log("Got data for '%s':", blobObj.name);
  console.log(buf.join(''));
  console.log("Metadata:");
  Object.keys(meta.metadata).forEach(function (key) {
    console.log(" * %s: %s", key, meta.metadata[key]);
  });
});
stream.end();

Every “end” event for both cloud requests and streams passes two parameters: results and meta. The meta object looks like:

{
  headers: {
    /* HTTP headers. */
  },
  cloudHeaders: {
    /* Cloud-specific HTTP headers (e.g., "x-amz-"). */
  },
  metadata: {
    /* Metadata headers (e.g., "x-amz-meta-"). */
  }
}

where the cloud-specific part (e.g., ”x-<PROVIDER>-”) has been stripped off as an abstraction. In this manner, you don’t have to worry if your request headers are formatted for AWS or Google – it just works.

Our end result from the request is:

Got data for 'foo.txt':
Hello there!
Metadata:
 * bar: 42
 * foo: My foo metadata.

If you have a blob object, the method is also available as Blob.get().

Blob

In addition to the aliased blob put() and get() methods from the previous section, Sunny provides:

There are also local file convenience methods:

Headers / Metadata

The above examples show custom headers / metadata that we can add in our options argument and retrieve from cloud operations in the meta parameter to our end event handlers. In both cases, the format is:

{
  headers: {},
  cloudHeaders: {},
  metadata: {}
}

So, for most cloud operations, we can call an operation like:

var requestOrStream = containerOrBlob.operation(/* args */, {
  headers: {},
  cloudHeaders: {},
  metadata: {}
});

We can also get back returned headers, metadata in the same format with the meta object:

requestOrStream.on('end', function (results, meta) {
  console.log("Non-cloud headers: %s", meta.headers);
  console.log("Cloud headers: %s", meta.cloudHeaders);
  console.log("Cloud metadata: %s", meta.metadata);
});

Errors

Sunny throws two types of errors: normal Error’s and Sunny-specific CloudError’s. Error objects are thrown when there is a programmer or library error (like a missing or invalid parameter during a method call) and should generally be tracked down and eliminated. For operations like putting a local file, etc., if the underlying operation throws a normal Error, then that can be passed back too (and should be handled).

CloudError’s should always be expected and handled by the caller. As all cloud operations are capable of throwing errors in various ways, the errors must be accounted for in program design. The CloudError API page describes the class in full, but here are some noteworthy points about the object.

CloudError

A CloudError has the following useful data:

and the following error type methods:

The methods correctly abstract differences in cloud providers, so should be used over status code / error message checks whenever possible. For example, if you try and create a container that someone else already owns, both Amazon and Google return a 409 status code, but Amazon’s error code is “BucketAlreadyExists”, while Google’s is “BucketNameUnavailable”. In both cases, Sunny allows you to detect this by calling CloudError.isNotOwner().

Error Handling and the API

Remember to check the API documentation for the cloud operation you are invoking. Many methods actually trap and swallow CloudError’s to provide a consistent, cloud-agnostic interface. For instance, creating a container that already exists will get a 200 “OK” response from AWS, whereas Google Storage will return a 409 “BucketAlreadyOwnedByYou” error. Sunny handles this specific situation by swallowing the error and returning a alreadyCreated boolean member of the results object on the “end” event callback.