Category Archives: Developers

Using Amazon SQS as a Messaging Bus

2 comments Published in Developers, Integrations, Tools and tagged , on by .

In a previous blog post I wrote about how we use a Message Queue server in a Message Bus design. Since that post was written we’ve moved from ActiveMQ to RabbitMQ and now we are moving to Amazon Web Service Simple Queue Service (SQS). I’m going to cover:

  • What is SQS?
  • The Message Bus design Pattern
  • How we’re using SQS to integrate with outside systems.
  • Bonus: I’ll also include how we’re looking at using SNS for notifications.

Intro to AWS SQS

Amazon Web Services are a suite of hosted services run by Amazon that are available on a pay per use basis. SQS (short for Simple Queue Service) is their messaging queue service. SQS operates much like other Queue servers, you can:

  • Create queues
  • Place messages on queues
  • Read messages off of queues.

One important difference with SQS is that there is no guarantee about the order that messages are delivered in. In most other stand alone queue servers messages are delivered in a First-In, First-Out (FIFO) manner. The primary reason that messages can be delivered out of order with SQS is a feature called message visibility. When a message is read from SQS it is no longer visible on the queue. This prevents multiple clients from getting the same message to process. But unlike some other queues, reading a message from the queue does not remove it. A client must explicitly acknowledge a message for it to be removed from the queue. If a client reads a message, but doesn’t acknowledge it, the message will become visible again a short time later. While the message is not visible other messages can be processed, and thus completed before the message becomes visible again.

What is a Message Bus

A Message Bus is a software design pattern that allows for message senders and receivers to be decoupled. This means that message senders and receivers can be added and removed without effecting other senders and receivers attached to the bus.

While we started with a message bus in our design for using SQS as a queue, we are not currently taking full advantage of the Message Bus design pattern as we use one sender class and one process for reading and distributing messages to the correct subscribers.

How we are using SQS

Now that we’ve covered SQS and the basic concept of a messaging bus, lets look at how we put it all together at MerchantOS.

Generating Messages

We have many different message types that are generated and placed on our queue (Item, Customer, Integration Specific messages, etc). Since we are generating all of these messages within our own application we have simplified our message generation code to one class that is used across our system. Using one central message producer allows us to reduce the overall architectural complexity of generating messages and placing them on the queue. This also allows use to integrate other message generators in the future as we need or want them.

Receiving Messages

On the receiving side we have used a few techniques to improve the reliability of receiving and dispatching of messages

Using a CRON to receive messages

To receive messages we use one CRON process that wakes up and forks child processes. These child processes are each responsible for taking one message off of the queue and forwarding it onto its appropriate destination. We have previously used separate processes for each endpoint that needed a message. The downside to this technique was again complexity of monitoring each process and diagnosing when issues arose.

Reducing Coupling

While using one parent process that creates worker processes has potential pitfalls as a point of failure, we have mitigated those by reducing the coupling of the worker processes to the endpoints that they call. One technique we used was to have the workers call the endpoint (typically an RPC endpoint on our API), post it’s message payload and disconnect without waiting for a response. Each endpoint becomes responsible for logging its success or failure. By keeping the workers short lived we reduce the overall impact of one worker failing.

Routing Messages

Another way that our queue usage has evolved is by consolidating all messages into one queue. Previously we used per account queues, which allowed for segregating accounts that might be generating many queue messages from those that only generate a few. In the long run we found that it failed to give us any practical benefit.

We now use a two queue system where all messages are first placed on a “priority queue”. These messages are read off and dealt with by the worker processes described above. If one account is generating many messages, or if we have determined that the endpoint they are calling is saturated and cannot process any more messages, they are re-routed to a delayed queue. Messages on the delayed queue are read once the message volume has reduced or the endpoint is ready to receive messages again.

This arrangement of a limited number of message producers, sending to a limited number of queues, being read by a limited number of consumers, has helped to reduce our queue complexity, improve the transparency of the message flow through the system, and given us better monitoring of our queue status.

Integrating SNS

Our next steps with SQS and our queue development is to create the ability to generate webhook callbacks that can be notified about messages on the queue. We are currently evaluating Amazon SNS among other options to provide a way to generate notifications. With a general notification framework in place it will allow for more complete workflows that include MerchantOS as one component, including potentially being able to create websockets for a richer client interface.

Wrap-up

Our queueing infrastructure has evolved over time to help us create decoupled integrations while reducing our application complexity. Amazon SQS is an integral piece in our ability to reduce our application complexity while also allowing us to scale as our user base grows and our messages per user increases. We’ve got some exciting changes coming to our queuing infrastructure that will allow for richer interactions with API clients. Watch our blog for more posts as we release these changes.

AWS DynamoDB For Session Redundancy And Failover

8 comments Published in Developers, PHP, Tools and tagged , , on by .

This article will go over what DynamoDB is and how we use it to backup our session data. It allows us to failover from one datacenter (AWS region) to another without losing session data and logging people out of our system.

Amazon Web Services (AWS) DynamoDB

DynamoDB is a cloud NoSQL database hosted by Amazon. You simply create a table and set the read/write capacity you want and Amazon takes care of the rest. No servers to manage or scale. Pretty awesome. Actually it’s so awesome that Amazon DynamoDB is the fastest growing new service in the history of AWS.

What’s is DynamoDB good at?

  • No hassle data store that is performant, scalable, and reliable.
  • Quick reads/writes. You decide the performance you need and pay for that.
  • Relatively cheap.
  • Access data via a key.
  • Store data that is in the 1-10KB range per item (you can store more but it gets expensive).

What’s does DynamoDB suck at?

  • It’s not a relational database. You can’t query it with joins or complex selects.
  • It’s not as fast as memcache (in our experience).
  • For large items or extremely high throughput it can get expensive (compared to running your own memcache service for example).
  • There is no simple way to back up your data (they do have a process by which you can get data to S3 but it’s pretty complicated and involves two other totally separate services from AWS).

Where can you read more about DynamoDB?

Using DynamoDB For Session Backup

In the blog post Scalable Session Handling in PHP Using Amazon DynamoDB they cover how to implement session handling for PHP using DynamoDB. I experimented with this, but what I found was it was too slow for our needs. Our session reads with DynamoDB were taking 20+ms. Our memcache session reads are an order of magnitude faster than that. We also already have memcache session handling implemented and working beautifully.

Why Do We Need To Backup Our Sessions?

For some applications it might be acceptable to log everyone out if you failed from one datacenter to another (heck lots of applications run in only one datacenter and don’t have any failover). Our application is mission critical for our customers – if we’re down they’re losing money. We run completely redundant setups in two different AWS regions on EC2.

If there’s a problem we have to switch customers from one datacenter to the other. Not having the sessions backed up in a way that both datacenters can access would mean everyone using our system would be logged out. This is a real hassle for our users because sometimes the stores are operating with a manager login and associates just have PINs to switch to their profile. If they get logged out they have to have the manager come by and log back in. What if the manager is out on lunch? Out of luck.

We previously stored our session backup in our MySQL database. But this became unscalable as the number of concurrent sessions grew. A few months ago we turned off our MySQL session storage and have been running just on memcache sessions. It improved performance, but it meant we might log everyone out if we had to switch datacenters.

DynamoDB To The Rescue

I looked at a number of different NoSQL type solutions for our session backup. DynamoDB made it to the top of the list because of it’s easy of management, scalability and price.

The basic concept is:

  • Session are stored and read from Memcache (every page hit)
  • Every 15 minutes we write the session to DynamoDB (each session stores its time since last DynamoDB write)
  • If we can’t find a session in Memcache (datacenter failure, or Memcache reboot) we look for it in DynamoDB.
  • Result: Users aren’t logged out if we switch datacenters or reboot Memcache.

DynamoDB PHP Code Samples

I gleaned a lot of this code from AWS blog post on PHP DynamoDB sessions.

readDynamoDB

This reads our session data out of DynamoDB when we need it. We call this from our custom session reading function (see session_set_save_handler)

function readDynamoDB($ses_id)
{
  $this->initDynamoDB();
  $result = '';
  $response = $this->_dynamodb->get_item(
    array( 'TableName' => self::DYNAMODB_TABLE,
           'Key' => array('HashKeyElement' => $this->_dynamodb->attribute($ses_id)),
           'ConsistentRead' => true, )
         );

  $node_name = 'Item';
  if ($response->isOK())
  {
    $item = array();
    // Get the data from the DynamoDB response
    if ($response->body->{$node_name})
    {
      foreach ($response->body->{$node_name}->children() as $key => $value)
      {
        $item[$key] = (string) current($value);
      }
    }
    if (isset($item['expires']) && isset($item['data']))
    {
      // Check the expiration date before using
      if ($item['expires'] > time())
      {
        $result = $item['data'];
      }
      else
      {
        $this->deleteDynamoDB($ses_id);
      }
    }
  }
  return $result;
}

writeDynamoDB

This reads our session data out of DynamoDB when we need it. We call this from our custom session reading function (see session_set_save_handler)

function writeDynamoDB($ses_id,$data,$expire_minutes)
{
  $this->initDynamoDB();
  // Write the session data to DynamoDB
  $response = $this->_dynamodb->put_item(
    array( 'TableName' => self::DYNAMODB_TABLE,
           'Item' => $this->_dynamodb->attributes(
             array( self::DYNAMODB_HASH => $ses_id,
                    'expires' => time() + ($expire_minutes*60),
                    'data' => $data,
                   )
            ),
    )
  );
  return $response->isOK();
}

deleteDynamoDB

This delete our session data from DynamoDB when we are done with it. We call this from our custom session destroy and gc function (see session_set_save_handler)

function deleteDynamoDB($ses_id)
{
  $this->initDynamoDB();
  $delete_options = array( 'TableName' => self::DYNAMODB_TABLE,
                           'Key' => array('HashKeyElement' => $this->_dynamodb->attribute($ses_id)),
                         );
  // Send the delete request to DynamoDB
  $response = $this->_dynamodb->delete_item($delete_options);
  return $response->isOK();
}

Developers: API 2.0 Change On Wed 8/8

1 comment Published in Developers and tagged , on by .

If you don’t know what an API is; ignore this and carry on.

If you’ve written an integration to the MerchantOS API or have had an outside developer write an integration take note! The API will change to version 2.0 this Wednesday night (8/8/12). Your integration may break if you are not prepared.

Here are the primary changes we are making in version 2.0 and how you can work around them to keep your current integration working:

  • shallow by default: Currently you can specify a shallow parameter on our API to have only the requested object, but none of its children or related objects returned. We are going to be setting this as the default behavior. To receive related objects you will need to specify a “load_relations” parameter with the related objects to load (see below).
  • readonly by default: By default all read (GET) requests will be using our readonly backend. For applications that are only reading or aggregating data there will be no changes. For applications that write data and then read that data out, be aware that there is a replication delay between our servers where the write requests (PUT, POST, DELETE) are processed and the servers where reads (GET) happen.
    • It is possible to override this behavior by setting the readonly parameter to false (readonly=false). Only do this if you need to read data immediately after writing it. Abuse of non-readonly reads can result in your application being rate limited or blacklisted.
    • Responses from writes are always realtime. When you write data you get the object that was created/updated back as a response. That data is always a real time response to your write. No need to worry about readonly=true.
  • load_relations search parameter: As mentioned above a new search parameter is being added that allows you to load related objects dynamically. Using this option you can load only the related objects that you need to get information from. All other related objects will not be loaded, allowing our API to respond to your request more quickly. load_relations=all will have the same behavior as the 1.0 version of the API.
  • JSON format changes: We are also improving our JSON emitter to improve its performance. In addition we are changing how single objects are returned in JSON format. Currently they are returned as bare objects. After the API version 2.0 changes they will be returned in an array the same way that multiple objects are returned. We hope that this change will simplify parsing JSON from our API.
    • We are also improving the layout of tags in our JSON responses so that they are not as verbose. Again we hope this will improve client parsing.

Workarounds and Fixes:

  • Version=1: The easiest work around is to set your version to 1. Adding a GET parameter of version=1 will accomplish this. You can also send a version header in the form of a specially formated Accept header like: application/vnd.merchantos-v1+xml (version 1 with xml output). Another example would be: application/vnd.merchantos-v2+json (version 2 with JSON output)
  • load_relations=all: Set load_relations=all as GET parameter to your query to load it with all relations just liket he default behavior in version 1.0. If you have an integration that needs to read a lot of data from us we recommend you do not use load_relations=all. Instead use load_relations=[set of relations you really need]. That will allow us to handle more of your requests.
  • readonly=false: If you need to write data and then read with those changes applied in real time set readonly=false as a GET parameter. As noted above this can get your application/IP rate limited or blocked if you abuse it.

OAuth 2.0

The API changes also include OAuth 2.0 for authentication. This allows us to set a per client rate limit. If you need a higher rate limit than 10 requests per minute you will need to implement OAuth 2.0 for your API authentication. We will then review your integration and increase your rate limit if you meet our quality standards.

Eventually we may phase out other authentication methods so it’s a good idea to implement OAuth 2.0 now. It also makes for a better user experience if you have end users setting up your integration (no more cutting and pasting API keys).

API Version 1.0 Sunset

We will sunset version 1.0 of the API when we release version 3.0. You should plan to make the changes necessary for version 2.0 compatibility even if you plan to use version 1.0 for now.

Why The Changes?

We’ve had some big challenges with our API around scaling. It’s easy for a programmer to write a loop and eat all our server resources. These changes are moving us towards a more scalable API.

Subscribe To The Developer Group

If you have an integration that uses our API, please subscribe to our Google Group.

MerchantOS API Changes

Leave a comment Published in Developers on by .

It’s been one year since we first introduced our API. It’s taken a lot of work to get where we are and it will take even more to get where we want to go with our API. We’ve seen ever increasing use of our API for external integrations like Perkville and internal projects like adding new inventory and generating packing lists. We wouldn’t have been able to make many of the improvements listed below without the valuable feedback of our API users. Thank you for your feedback and suggestions.

We are proud to annouce some changes and improvements to our API that are coming in our next release. Here is a quick overview of what is coming.

API Errors Returned in the Requested Format

Now when the API encounters an error instead of returning an html error page it will return error information in the format that was selected in the request (XML or JSON). This should allow for better error handling by client applications.

Fixes to API Datetimes

There was a bug that was causing some datetimes in the API to be returned incorrectly. Datetimes are now returned as GMT corrected for the timezone set in account.

Add Category and Tags to Receipt Template Items

We had a request to load the Item Category and Tags on the XML returned in the receipt template. This change will give access to those values to people using custom receipt templates.

Load Customer and Employee on Workorders

Another request we had was to load Customer and Employee information on returned Workorder objects. This gives access to the Customer name and address and Employee details for a Workorder without having to make more requests.

Sale Write Access

The biggest change that we have made is open up write access to Sales via the API. This is an important step forward for our API as it exposes more of the functionality of our application via our API. There are some important caveats to the functionality that we have exposed:

  • Refunds and Returns are not yet supported
  • Sales can be Archived (using the Delete method) or Completed, but they cannot be Unarchived or Uncompleted
  • We have made every effort to fully test Sale write access via the API, but proceed with caution. If you find an error or discrepancy please contact us and let us know so we can correct it.

We still have plenty of work to do to improve our point of sale software API, including improving performance and exposing write access to more objects. We’re excited about these changes as they allow us to make big improvements to our product in the future.

If you have an API question, comment, suggestion or issue feel free to contact Nate at nate@merchantos.com.

API Updates

Leave a comment Published in Developers, New Features on by .

We’ve been getting lots of great feed back about our API. There are several projects that have been using it and we’ve been working with them to answer their questions and solve some problems that they have been running into.

In our latest release we’ve made a few adjustments to how the API works. None of these changes should break compatibility.

Reduced API Rate Limit Penalty

We had a penalty included in our API for when an application went over their call limit. We had found that this penalty was too harsh and we’ve reworked it to allow applications using our API to hit the call limit occasionally without penalizing them.

Return 503 When Over Rate Limit

Previously we had been returning a 500 error when an application went over it’s call limit. We’ve changed that to a 503 Service Unavailable error as that better reflects the error.

JSON input and output

We’ve also been working on adding JSON input and output for requests. Previously we could output JSON, but now we can accept JSON for requests that require a request body.

Disabled WWW-Authenticate header for AJAX Requests

If you are sending an AJAX request we will no longer be sending the WWW-Authenticate header as this causes UI issues with browsers. All other clients will still receive the WWW-Authenticate header when their request could not be authenticated.

Setting a Default Limit on Listing Requests

Previously we did not page results by default. This was causing issues with very large result sets. Now listing requests (such as requesting all items) will by default return 100 objects per request. This behavior can be over-ridden using the limit and offset query parameters (see the API search manual page: https://manual.merchantos.com/index.php/API_Searching). We are also returning the total count of objects, the current offset and limit for a request as attributes on the outer object tag when we automatically limit a request.

Tagging

We have added the ability to write tags through our API. Objects supporting tags (currently only Items) will have a Tags element containing the tags on the object if there are any. The count of the tags will be returned as an attribute of the Tags element.

Tags can be added to an Object by adding xml similar to below:

<Tags>
    <Tag>
        <name>foobar</name>
    </Tag>
</Tags>

The tag does not have to already exist in the account you are accessing, it will be generated when your object is submitted.

All Tags can be accesses through the new Account/<AccountNumber>/Tag endpoint. This will show you how many tags there are as well as what objects are associated with that tag.

Bug fix to IN queries

We fixed a bug that was preventing queries with the IN search operator from executing properly. Queries like the following should now execute:
https://api.merchantos.com/Account/<AccountNumber>/Item?itemID=IN,[2,4,6,8]

More Improvements to Come!

We’re open to your feed back. If you have any questions about the changes we’re introducing please feel free to contact us. If you are having a problem with our API let us know and we will work with you to resolve it. Thanks again to all of the developers using our API for their valuable feedback. We’re working on even more improvements and we will post again when they are released.

Developer Digest

Leave a comment Published in Developers on by .

This month we check in with Vice President of Engineering Nathan Horter. Nate’s primary job is to make sure that your vendors are seamlessly integrated with MerchantOS, so as to cause you as little pain as possible. Nate also works on the back end infrastructure of our software, ensuring that everything is running smoothly behind the scenes.

What are you working on lately that customers might be affected by?

I’m currently working on overhauling our Shopify integration to improve it’s efficiency and squash some bugs. There shouldn’t be a huge difference in functionality once this update is released, but we will have solved most of the bugs that people have reported to us about the Shopify integration.

I’m also working on fleshing out some missing functionality in our API and starting on implementing webhooks. This will add Sale write capability to our API, which is a much needed feature for several integrations people are working on. Webhooks will allow our users to send notifications from our system to other web based services when certain actions are taken (like when an item is saved, or a transaction is completed, etc).

What’s your big idea for the future (that you’re willing to share)?

The biggest thing we’re working toward right now is separating the backend data (the numbers, and information stored in our system) from the front end representation (the experience that our users have interacting with the system). This allows us to be more flexible in creating our front end design and gives our users the opportunity to adjust the front end to their needs more easily. It’s a slow process, but it means that our software will become more scalable, flexible, and robust.

What is one thing you want every customer to know about you?

I’m always interested in helping our users get their vendors integrated with our system. It usually works best if a retailer talks to the vendor about integrating with MerchantOS. It’s difficult for us to get our foot in the door and convince them that integrating is in everyone’s best interest. Having a retailer, who does business with them, ask them to integrate makes more of an impact. If you’ve talked with a vendor and they are interested in integrating, have them contact me at nate@merchantos.com or 866-554-2453 ext. 95.

Ivan the Developer

Leave a comment Published in Developers and tagged , , , , on by .

This month we catch up with founder and CEO, Ivan Stanojevic to get the inside scoop on the behind the scenes work he does to keep MerchantOS software ahead of the competition.

First off, what hats do you wear when it comes to software development?

IVAN: My two main roles with the product are making sure it’s always available to our customers, and keeping each new release in line with our vision. I don’t spend any time writing code directly used in our software.

We have a growing infrastructure of servers that provide our web based service to our customers. I keep that infrastructure healthy and get our content delivered to our customers reliably and fast. Most recently I have been working on adding in a new data center and preparing to use some newer technologies to speed up our systems. We’re also scaling up our infrastructure with more capacity.

When working on the actual point of sale system, I spend a lot of time reviewing support requests to see where our software can improve, help the developers decide what to do next, test out new features and fixes, and make sure those changes both meet our customer’s needs and are simple to use.

What have you got going on right now?

IVAN: Right now I’m working on adding in a new data center with Amazon Web Services. This will allow us to scale our infrastructure much more quickly and save us the hassle of managing our own hardware.

This change should have no effect on our customers at first, but will allow us to speed some things up later.

What’s your big idea for the future?

IVAN: I don’t really have any one big idea. There are tons of changes we have to make to keep our system innovative and further improve it’s simplicity. We’re going to just keep rolling out the improvements.

“I don’t think our software will ever reach a point where it couldn’t be better.”

There are many new technologies coming along that can change how retailers are able to serve their customers. Two active areas I see are with mobile and payment processing.

We’re definitely going to be spending more time on mobile devices and you’ll be seeing those changes coming along.

We’re also watching the payments industry closely. I am hoping for some disruptive technologies that can compete with and change the traditional credit card industry into something that better serves retailers. As we see great ideas emerge we want to be ready to jump in and use those technologies in our product.

Before we go, what is the one thing you want every customer to know about you?

IVAN: I’m willing to talk to any customer. Email me at ivan@merchantos.com or call at 866 554 2453 ext. 91. I can’t promise a quick response but I do eventually get to everyone.


Related Articles

Using ActiveMQ for Web Services Integration

2 comments Published in Developers and tagged , , , on by .

UPDATE (3/26/13): MerchantOS no longer integrates with Freshbooks. If you need support, please call 866-554-2453.

If you’ve been reading our Blog over the last few months you’ve read about the integrations we’ve been creating with other web based applications like Freshbooks and MailChimp. These integrations allow our customers to use high quality web based applications with their data from MerchantOS and allow us to focus on making our application better.

Getting two completely different applications to talk to each other is no easy task. And we want MerchantOS to talk to more than just one other application. We needed a way to make creating new integrations easy and scalable.

We solved this problem using an idea called a Message Bus. In software design a message bus is a common connection among many programs we can send messages over. We’ve implemented our Message Bus using Apache ActiveMQ. Using this Message Bus layout we can create topics, send messages to those topics and have all (or some) of our integrations read those messages.

This means that you can, for example, create a customer in MerchantOS and have that customer’s information automatically exported into FreshBooks and MailChimp. And since we track where we have sent that customer’s information we can update it when you change it in MerchantOS in the future. This type of integration saves you time since all of your data is updated at once. But this is only a simple example, what about a something more complex?

We have an integration with Shopify that goes a step farther than just being able to export data from MerchantOS. With our Shopify integration orders that are placed in your Shopify shop will automatically be imported into MerchantOS. This automatically creates a new invoice and transaction.

This is a powerful feature by itself, but it’s power is multiplied when you have an integration with FreshBooks and MailChimp. With all three of those integrations turned on in MerchantOS:

  • MerchantOS receives an order from your Shopify account and creates an invoice in MerchantOS.
  • MerchantOS then exports that invoice to FreshBooks so you can email it to the customer .
  • MerchantOS also exports the customer’s information to MailChimp so you can send that customer emails about specials or other notifications later.

All automatically.

We’re going to be adding more integrations soon, so let us know what you would like to see us integrate with next!

Challenges For Our iPad Point of Sale Dream

24 comments Published in Developers, News on by .

I’ve spent a couple weeks tinkering with the iPad and the end result has been frustration.

What is possible right now

If you’re willing to use the onscreen keyboard to search for items or if you can use our categorical menu (check out the video), you’re going to be able to have a device you can sell product with while roaming the shop with your customers. If you need bar code scanner compatibility, the iPad isn’t quite there yet. You can also plug in either a USB barcode scanner or USB credit card reader via the camera connector kit but you will lose onscreen keyboard functionality while that device is plugged in. This really hinders the efficiency of using the iPad for now.

Here’s a list of issues I couldn’t work around

  • If card reader or scanner are plugged in, no onscreen keyboard pops up.
  • Can’t charge iPad while any other USB device is plugged in. I tried using this iPad mount kitmade by ram-mounts. This mount system is hands down awesome! It uses a ball connector allowing you to swivel the iPad around however you want and there are many mounting options. Sturdiness was certainly good enough to use in a retail environment for the mount setup I used.

    There is some hope

    iOS 4.2 appears to be adding in some powerful features including the ability to print to a remote printer. It is possible though not likely this update might allow USB devices to function better.

    And even more hope…

    Mobile device support is becoming more of a priority as we develop our system. We plan to experiment with more tablet style devices and, over a long period of time, officially support mobile devices by creating browser based pages designed for those devices.