S3 | NBOS TECHNOLOGIES

Are you planning to move your existing GB’s or TB’s data to s3?
Below nodejs script might help you to get started and upload your data to S3.

PreRequsite:

Ensure that aws-sdk node module is configured. If not follow AWS documentation https://aws.amazon.com/sdk-for-node-js/

npm install s3 --save

1	npm install s3 --save

s3 Options:

Usually accessKeyId and secretAccessKey is enough to make it happening. But if you are behind proxy, use those commented s3Options fields accordingly.

Source Directory

localDir: "<localdir>",

1	localDir: "<localdir>",

For windows directory delimiter needs tobe delimited, eg c:\\mysite\\uploads

For Linux you can use something like ‘/var/www/site/uploads’

var s3 = require('s3');

//s3 client
var client = s3.createClient({
  maxAsyncS3: 20,     // this is the default
  s3RetryCount: 3,    // this is the default
  s3RetryDelay: 1000, // this is the default
  multipartUploadThreshold: 20971520, // this is the default (20 MB)
  multipartUploadSize: 15728640, // this is the default (15 MB)
  s3Options: {
    'accessKeyId': "<accesskey>",
    'secretAccessKey': "<screct access key>",
    //'sessionToken': "<use only forPrivateVPCaccountsonly>",
    //'httpOptions': { 'proxy': 'http://proxy' },
    //'sslEnabled': false,
    'region': 'us-east-1'
    //endpoint : "some.endpoint.com.s3-website-ap-southeast-1.amazonaws.com",
    // s3BucketEndpoint:true
    // any other options are passed to new AWS.S3()
    // See: http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Config.html#constructor-property
  },
});


var assets = {
  localDir: "<localdir>",//windows: eg d:\\uploads, unix: /var/www/site/uploads
  deleteRemoved: true, // default false, whether to remove s3 objects
                       // that have no corresponding local file.

  s3Params: {
    Bucket: "yourBuckerName",
    Prefix: "folderA/",
    // other options supported by putObject, except Body and ContentLength.
    // See: http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#putObject-property
  },
};

var assets_uploader = client.uploadDir(assets);

assets_uploader.on('error', function(err) {
  console.error("unable to sync assets folder , error :", err.stack);
});


assets_uploader.on('progress', function(err) {
  console.log("Processing the data sync for the direcotry assets ." , assets_uploader.progressMd5Amount,assets_uploader.progressAmount,assets_uploader.progressTotal)
});

assets_uploader.on('end', function(err) {
  console.log("Done uploading the direcotry. assets !!!")
});

var s3 = require('s3');

//s3 client

var client = s3.createClient({

maxAsyncS3: 20, // this is the default

s3RetryCount: 3, // this is the default

s3RetryDelay: 1000, // this is the default

multipartUploadThreshold: 20971520, // this is the default (20 MB)

multipartUploadSize: 15728640, // this is the default (15 MB)

s3Options: {

'accessKeyId': "<accesskey>",

'secretAccessKey': "<screct access key>",

//'sessionToken': "<use only forPrivateVPCaccountsonly>",

//'httpOptions': { 'proxy': 'http://proxy' },

//'sslEnabled': false,

'region': 'us-east-1'

//endpoint : "some.endpoint.com.s3-website-ap-southeast-1.amazonaws.com",

// s3BucketEndpoint:true

// any other options are passed to new AWS.S3()

// See: http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Config.html#constructor-property

});

var assets = {

localDir: "<localdir>",//windows: eg d:\\uploads, unix: /var/www/site/uploads

deleteRemoved: true, // default false, whether to remove s3 objects

// that have no corresponding local file.

s3Params: {

Bucket: "yourBuckerName",

Prefix: "folderA/",

// other options supported by putObject, except Body and ContentLength.

// See: http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#putObject-property

};

var assets_uploader = client.uploadDir(assets);

assets_uploader.on('error', function(err) {

console.error("unable to sync assets folder , error :", err.stack);

});

assets_uploader.on('progress', function(err) {

console.log("Processing the data sync for the direcotry assets ." , assets_uploader.progressMd5Amount,assets_uploader.progressAmount,assets_uploader.progressTotal)

});

assets_uploader.on('end', function(err) {

console.log("Done uploading the direcotry. assets !!!")

});

Use Case:

A PHR(Patient Health Record) application gets health reports of patient from various hospitals and labs. Patient Health records are in XML format which needs to be processed and stored into Database.

Lets see how we can use Amazon AWS to implement this use case.

Step1: Create a Amazon Simple Storage Service (S3) bucket where documents to be uploaded.

Step2: Create a AWS Lambda Function using NodeJs. Configure Lambda function to get triggered on XML upload to S3. Check below screen shot.

Step3:

Create a Mysql Database using AWS Relational Database‎ (RDS).

Step 4:

Lambda NodeJs code can be dowloaded from Wavelabs Git Repo.

Package the JS file into zip file and upload the NodeJS Lambda function. Make necessary changes in index.js file as per your requirements.

Ouput:

From AWS dashboard, upload sample xml file( use file patient.xml from repo) to s3 bucket.

Connect to Mysql using command line or using workbench and see the results.