One of the challenges with a high level of automation is that if one critical cog in the machine is misaligned,  it can create problems that may not show up until weeks or months later.
For UBO 6 Beta, certain exceptional sets of variables have resulted in errors in a small percentage of invoices over the last few months. We’ve manually repaired the invoices, and systematically repaired the code. We’ve also identified the common-denominator culprit, which has been edits to the Paid Thru date by innocent, well-intentioned app users. Unfortunately, at this point, so many automation functions are tied to the Paid Thru date that those edits can have unexpected consequences for that subscriber’s future automated invoicing.

To resolve these issues, we are doing the following:

  • We’ve engineered a detailed map to navigate every known possible combination of billing scenario in Version 6.
  • We’ve developed an exhaustive testing protocol for invoice changes and have executed the protocol on our “Time Travel” test server where we can prove in a matter of hours the invoicing functions as expected over a period of many months, by essentially forwarding the clock of the server and executing the scheduled auto-actions and crons.
  • Every possible scenario where editing the paid through date has been required has been mapped out so that editing the paid through date will be unnecessary and the ability to edit the paid through date will be eliminated.  The paid through date will be editable by our staff only to be sure it doesn’t cause unexpected behavior in your billing.
  • Version 6 will not be released out of beta until we’re absolutely confident that there are no issues and the bug count (which is already at a record low) drops to and stays at zero for some period of time.

Update 11/28/2012: Paid Thru date changes are an external variable and just one piece of this puzzle.  The remaining invoices affected are roughly 1 in 1000 where a few ISPs have all the right variables and have had multiple invoices affected and most other ISPs have been completely unaffected.  We’ve written auditing tools to identify every affected invoice and are currently proactively correcting the errors.  If you have affected invoices, we’ll follow up with you directly with full details. To significantly reduce the likelihood of these kind of issues appearing again, Version 7 invoicing code has been completely redesigned from the ground up to eliminate Paid Thru date dependencies and complex variable relationships.
– Todd Grannis, CEO