We’ve long been fans of Zapier and over recent years many customers have made lots of really useful integrations. The combination of agileBase and Zapier allows you to not only develop back office systems quickly but also link them into the wider ecosystem of cloud applications, all without writing a single line of code.
We’re now improving the Zapier integration to allow even more use cases using push.
The standard way to link Zapier to agileBase is to create a Zap based on a webhook poll. Every few minutes, Zapier will poll agileBase to see if there’s any data to send over. The most frequently this can be set to happen is once every 5 minutes.
This works well for cases where the action’s not particularly time sensitive, such as sending new email addresses to Mailchimp or invoices to Xero for example. However sometimes, you (or rather the user) want the action to take place immediately, on the press of a button.
agileBase has had the facility to push data to third party systems on demand for quite a while, but up until now these haven’t worked well with Zapier, rather requiring manual integration by a coder.
Now, some options have been added to the Push API settings to work with Zapier out of the box. Furthermore, these options will default to Zapier-compatible settings for new integrations.
In Zapier, set up a webhook integration and select ‘Catch Hook’ or ‘Catch Raw Hook’.
In agileBase, paste the URL Zapier provides into the Push URL field. For Push data format, select ‘JSON content body (Zapier compatible)’ and optionally tick ‘generate simple format JSON’.
A more detailed tutorial on how to use Zapier with agileBase will be coming soon, in the meantime, please get in touch if you have any questions when you try this out.
Thanks to our partner Little House Consultancy for suggesting this improvement and our customer Lewis Pies for being the first people to beta test and try it out in anger.
A couple of other platform improvements have made the cut for today’s release:
Floating help panels
A great suggestion from our next-door neighbours at the Bristol Bath Science Park, Willow DNA, who are expert e-learning consultants – thanks!
When you click a question mark icon on screen to show the relevant help for a field or block, the help popup can now be dragged around the screen. You can type and use the form on screen as normal while the help remains visible. That means you don’t have to look at the help, try to remember it then close it before acting on it.
Views with charts are shown
If a view has a one or more charts, a chart icon now appears next to the view name in the menu. Hovering over the icon shows the names of all the charts.
That saves the user having to know or guess which view contains the charts they want to look at.
As a reminder, views can be organised into groups by adding the group name into the viewname, with a dash separating them – see the last part of this blog post. If you’ve already set up groups, these will carry through into the new interface.
Any views which are not in a specific group will be placed in a new group named as per the tile.
The explanations which appear under each view name come from the view descriptions as entered in the admin interface. Go to the manage tab of a view to edit its description.
Controlling hierarchy view
The hierarchy header is displayed when a record contains a relation to another, however as an administrator you can control when this happens.
To make a relation appear as a ‘parent’ of a record, simply move it up to the top of the screen, in the fields tab of the table. Similarly, move it down to take it out of the header area. It will be displayed as a parent if it’s the first field in the table, excluding special fields: cross references, titles, images and auto-generated number sequences.
Over the summer, we’ve been concentrating on making the agileBase user interface the best it can possibly be. We’ve run UX workshops, organised testing by people who’ve never used the system before and brainstormed a number of design ideas.
In August we introduced many small tweaks to improve the friendliness of the system. Now we’re ready to push the envelope and release two more significant updates, available from tomorrow.
The first thing you may notice on logging on and opening up a tile is that we’ve improved the navigation menus near the top of the screen, making it much easier to find things.
Clicking on one of the menu headers will open up a dropdown list of menu items underneath it. Each item consists of a view name, a count of records (if under 50) and a description of what that item contains, if necessary. That provides more context for the person in front of the screen and makes more efficient use of the space above the main contents of the page.
Data in your system is stored in what’s called a ‘relational database’, which means that records of different types can link together.
For example, a company record may have a list of contacts related to it. When editing a company, you can typically see the links to contacts in a tab, so the relationship is obvious.
However in the contact record, it has not (up until now) been visually clear that there is a company ‘above’ the contact. In other words, that the contact is a ‘child’ record of the company.
That connection to a hierarchy has now been made obvious with the addition of a heading line at the top of a child record linking to the parent(s). Here’s an example, where you can see that the contact Kevin Williams works for Stapleton Spices.
We’ve used companies and contacts as an example, but the same thing goes for any two types of data which may be related, for example customers and orders, projects and tasks, invoices and invoice lines or staff records and training records.
Clicking the relation link (Stapleton Spices circled above) will take you directly to that record.
If the link is wrong, e.g. if Stapleton Spices has been connected to Kevin by mistake and he should actually be linked to Stapleton Herbs, you can hover the mouse over the link for a few seconds and the option to change it will appear.
If a parent hasn’t been chosen at all, an input box will be displayed rather than a link, letting you type in or choose one.
This is a significant update to the software and we think it will make a big difference to people, helping them understand where they are in the system at a glance.
A post for system administrators will follow shortly, explaining how to customise these features in your own system. In the meantime if you’ve any questions, please do get in touch.
Recently a partner said to us ‘You know, some of your customers are actually using agilebase as an engine to power their business, in ways that go beyond simply building applications.’
And it’s true, agileBase is being used not just to deliver user-facing apps, but to act as a central store, processing data which is fed in and out of multiple third party systems, including for transaction processing, interacting with machinery and feeding public websites. It’s a service upon which other elements can build.
Emergent use-cases like these occur so often with startups: customers find new and exciting ways to utilise the technology which the founders didn’t necessarily envisage. This in turn opens up opportunities to grow the business in new ways and to provide way more value than ever before.
This use does however place more demands on people and systems, covering many facets, including
Scale: the throughput of data possible with API-based inflows and outflows is much greater than that of human beings manually interacting with systems. The same goes for automated workflows which can act on many thousands of records per second
Criticality: if a user can’t do their job for 5 minutes because the system is down or broken, that can be critical, but if an interaction with a third party system fails for 5 minutes causing multitudes of transactions to fail and potentially reputational damage, that’s much more likely to be critical. Both of these events have very low probability but still the consequence of the latter is far greater.
Complexity: when agileBase transitions from a platform powering user-facing apps to an engine powering a network of interdependent and often third party services, system complexity increases, so new means of dealing with this need to surface.
We recognise this is a major change. If growth in this area continues at its current trajectory and nothing is done, the system will start to be overwhelmed, with all customers being affected.
However, that’s not going to happen as we’re not standing still. We’ve already started implementing concrete changes to ensure we’re ahead of the curve. We’re serious about planning for and supporting this use of the platform, as opposed to just capping usage at a certain number of requests per minute or limiting complexity artificially.
Work we’ve started
At a board level we have agreed to commit to support the option for customers to use “agileBase as an engine”, in other words following the principles behind Service Oriented Architectures.
As a direct consequence of this we have engaged consultants to help us identify how we need to adapt, both technically and commercially.
Technical – platform development
Some immediate recommendations we’ve actioned or planned are
Employing people to increase the test coverage of
all of which must, given the increased criticality of the use case, have a higher level of stability and predictability. We’re also
Updating development processes to increase automation and reduce errors. (Amongst the tools in our armory are continuous integration, unit tests, static analysis and 3rd party library auditing)
Updating deployment processes to minimise downtime
Taking measures to shape API traffic, to avoid usage spikes from one source impacting the whole system
Introducing new workflow features (see below) to make them even more powerful and useful for automations
In the longer term, we’ve a roadmap which goes much further, with items such as
completely removing the need for any downtime at all when deploying new versions
scaling horizontally for virtually unlimited capacity increases
Customers will now be able to run a much higher volume of workflows and rely on larger volumes of transactions flowing through the API, which opens up new ways of using the platform.
We will in future be introducing two new pricing elements to accommodate this, one for workflows and the other for APIs. These will allow us to
ensure the platform can move forward at the required pace
properly service and support customer requirements in the professional ways expected of us.
Please be assured that general base costs will not be rising, only customers using these particular features will need to factor them in.
We greatly appreciate the innovation that current customers have brought to the table by using the system in this novel way. It’s what helps push us, and everyone else, forward.
Therefore we want to recognise this value and help these companies by offering a free remote session or face to face meeting, to
Uncover your most important mission critical elements so we can, at a later date, build customer-specific unit tests to cover these elements to add to our growing test suite
Gather further feedback and requirements about what would make the system even more useful, with the results being prioritised in our roadmap
We will be getting in touch with any customers who may benefit from this, and to assess whether there are any cost implications for the future (there will be no immediate changes).
As always, we look forward to talking to you. Get in touch by any of the methods listed on our contact page, or drop in and visit us at Bristol & Bath Science Park.
Background to Workflows and APIs
Fundamental to many of these new and innovative usage patterns are two features – workflows and APIs.
The thinking behind the development of these features followed the same pattern we’ve used while creating the whole agileBase platform; make things which are powerful because they’re easy to use (following common principles) and flexible i.e. generic enough to be used for many different purposes.
For the technically minded, APIs can be auto-generated for any View in the system, by ticking a box. Developers get an industry standard Restful, JSON based API based on the contents of the view, complete with https://swagger.io/ description for easy import to third party systems.
Similarly, workflows are built on Views. Using the built in agileBot you can set rules for how to automatically process data – whether creating, updating or deleting it, as well as generating documents and automating email communications to and from people.
Example: one customer running a membership franchise uses workflows to automatically recognise and de-dupe leads coming from multiple sources, to aid the attribution of leads back to marketing campaigns. APIs then send and receive data from the website to track customers’ touch points and spending.
To help customers with this particular emergent usage pattern, we’ll be introducing enhancements to both of these features, so watch this space.
Today we’re taking the unusual step of announcing details of an upcoming release in advance. It’s a feature packed release with some significant updates.
The first thing you’ll notice is the new icons! We’re using Duotone, a beautiful new icon style available as part of Font Awesome Pro. As well as the stylish new look, there are more icons to choose from for app developers. Our favourite is ‘pepper-hot’:
We have also changed the way the Tiles on the Home Page display to give you more control. The order can now be fixed by by going into the “Tiles” area of the administration interface and using the numbering.
Tiles will still change size depending on how often they’re opened, but the least used won’t move to the end of the screen. It was a nice idea but actually, testing has shown that the system’s easier to use when they remain in the same positions. Maintaining a logical order is now possible, so you can put suppliers next to customers or ingredients before products.
User interface tweaks
Many other elements of the system have been tweaked, following analysis of our most recent UX testing programme. For example, number prefixes in dropdowns have been removed (the numbering is used purely for ordering purposes) and there’s a new UI to show sub-tabs.
Even more UI changes are planned, so watch this space!
When using inline editing in a tab, e.g. as in the screenshot below of editing a recipe, all calculations will update as you type. In this example when you change an ingredient quantity, all the percentages update as well as the quantity total at the bottom of the list.
On demand workflows
Until now you’ve been able to set up automated workflows to run regularly, up to every 5 minutes, or on creation of a new record. Now you can also run them every time a record is loaded by a user. Since you can do so much with workflows, like edit records based on rules, or generate documents, this gives you the ability to set up automations to work in a much more user friendly way, without people having to refresh records or wait for changes.
As we say, there’s lots more in the pipeline and we’ll shortly have some news to share about a new type of usage pattern for agileBase. In the meantime, we hope you enjoy the new user interface which will be released in the next few days – we always aim to keep the system fast, friendly and flexible.
Near the end of last year, we introduced export notifications for sensitive commercial or personal data. Along with various other features, that lets administrators of a system be notified by email if a user exports more than a certain amount of data.
Well, this week’s update expands on that. As well as being notified, administrators can, per table, control how much a user can export at any one time, i.e. they can set a hard limit. This limit takes effect per export, so bear in mind it wouldn’t stop someone exporting an entire database of contacts for example, by doing lots of different filters and exports one at a time (e.g. all As, all Bs, all Cs etc.) but it would increase the effort and the administrator would get lots of notifications.
To set this up, go to the Build section in the agileBase admin interface. Then go to the manage tab for the table you’d like to control.
Make sure that one or both of ‘stores sensitive’ or ‘stores personal’ data are ticked and you’ll see two options. The first lets you set the limit at which admins will be notified of exports (e.g. 10 records). The second, which is the new option, lets you set the ‘hard’ limit. If someone tries to export more than this, the rows exported will automatically be cut off after this number.
The export limit applies to all views based on the table, in fact all views which any data from the table appears in. If a view uses data from many different tables, the minimum export limit will be applied when exporting from that view.
Please take a moment to think whether this will be useful for your organisation and set it up if so – we want all customer data to be as well protected as possible.
If you’ve any queries, don’t hesitate to get in touch as usual.
Last week we started talking about scalability in terms of the API and that got me thinking about what we’d do if and when we need to process orders of magnitude more data than we do currently.
Currently, the largest single table of data any one customer has comprises just over thirty million records. That may be a lot of business data in some contexts but in database terms it’s hardly ‘big data’. What about if we needed to store and work with billions of records?
A natural direction to look in would be to www.citusdata.com. As you may know, agileBase uses the open source PostgreSQL database. Citus transforms PostgreSQL into a distributed database. And it’s open source too. Their pitch is ‘Never worry about scaling again’.
Citus has recently been bought by Microsoft and is now available as Hyperscale (Citus), a built-in deployment option for Azure Database for PostgreSQL. It can also be used on AWS or your own hosting. So there are many options to allow scaling to many nodes to handle billions or hundreds of billions of rows, while maintaining high performance.
That’s for the future, but if anyone has an inkling of any projects they’d like to put forward which deal in high volumes of data a.k.a. ‘big data’ in the IT world, then drop us a line. We’d be keen to be involved.