From movement to figure - what happens between sensor and dashboard

Between a person walking through a door and the figure on a dashboard there are about ten steps. Any one of them can distort the figure without making it look wrong.

Most vendors describe this path in a single sentence: the sensor counts, the platform displays it. That is about as accurate as “the aircraft flies” is as a description of aviation. If you are buying a system whose figures feed budgets, staff rotas and reports to the board, you should know what happens in between - because the answers to the right questions differ a great deal.

This article walks through the whole chain once. Many of its links are covered in detail in earlier articles. Here they appear in order for the first time, and four that have not been explained anywhere yet get a section of their own.

The chain in ten links

The order is not arbitrary. Checking comes before counting, and counting comes before anything else is derived from the message. The sections below show why that matters.

Link 4: why the limit sits on the bytes

Every message on the network announces its size in its header. An obvious safeguard would be to check that figure and reject anything too large. The problem is that the figure comes from the sender. A faulty device or an attacker can announce a small size and send a large message.

The only limit that holds is one applied to the bytes actually received. On top of that comes a check of the message's structure against a fixed schema - with upper bounds on how many frames, events and objects a single message may contain. Anything that breaches these limits is rejected before any processing time is spent on it, and answered with a clear error code.

That sounds like security engineering, and it is. But above all it is data quality: a message with the wrong structure is not just suspicious, it is also useless as a figure.

Link 6: why a counter must not read before it writes

The obvious way to increase a counter has three steps: read the current value, add one, write the new value back. That works as long as only one message arrives at a time.

At a site with several sensors, though, messages arrive at the same moment. Then this happens: both read 11, both calculate 12, both write 12. Two people walked through the door; one was counted. The error shows up in no log, and the result looks plausible - it is simply too low.

The fix is a single write that hands the increase itself to the database: “increase by one” instead of “set to 12”. The database carries it out as one indivisible operation, so parallel messages can neither overwrite each other nor count twice. The term is atomic increment, and whether a vendor writes this way belongs in every technical review.

1,440 minutes instead of 34,560 messages

A sensor reports every two and a half seconds. If every report were stored as a record of its own, each site would produce 34,560 entries a day. Instead, writes go into one counter per minute: 1,440 records a day, around 96 per cent fewer, with no loss of minute resolution. With several sensors, there is one counter per sensor and minute, so each one can be checked on its own.

The minute is a deliberate choice. It is fine enough to tell a peak at 12:07 from one at 12:15, and coarse enough for an analysis across a whole year to stay fast. Which resolution an analysis shows, from minute to day, is set by the server according to the period - not by whatever tool retrieves the data.

Link 7: when a write fails

Databases are occasionally unreachable for a moment. The question is not whether that happens but what becomes of the figure when it does. There are three possible answers. It is lost. It is retried at once, and if that fails too, it is lost. Or it is kept durably and made good later.

Only the third holds up. The mechanism is called a dead-letter queue: a failed write lands in a durable queue and is retried up to three times, at growing intervals of around 5, 10 and 20 seconds. The interval carries a random variation of 20 per cent. That is not a flaw but the point: without it, after a brief outage every failed write would come back at the same instant and overload the database the moment it recovered.

There is one more detail, relevant only to platforms that run on serverless functions. Such a function can be frozen after it has sent its response, in the middle of work it has not finished. An operation that has therefore been marked “in progress” for more than five minutes is released automatically and tried again. Without that safety net it would sit there forever.

Because the duplicate claim sits in front of the write, every one of these retries is harmless. A message tried three times still counts once.

Link 8: one message, many analyses

The message that increases the minute counter contains more than a number. After the count, it produces dwell time per zone, paths through a space, view direction, speed and body height, queue situations, the evaluation of alert rules, and the calls to displays and building systems.

The order is what matters. All of this is produced only after the count has been safely written. A heavy analysis that takes longer or fails can never drag the base figure down with it. The count is the foundation; everything else stands on it.

How to read these derived measures is covered in the articles on each: zones and dwell time in Measuring dwell time, queues in Managing queues, demographics and objects in What a counter can tell apart, alerts in From number to action.

What makes a chain reliable

Put the ten links together and four properties remain against which any platform has to be measured:

• No count is written twice, even if a message arrives twice or is retried.
• No count is overwritten, even if several sensors report at once.
• No count is lost silently, even if the database fails to answer for a moment.
• No analysis looks better than it is, even if a sensor half fails or the clocks change.

The first three concern writing and are described in this article. The fourth concerns reading and is covered in The half outage. Both halves are needed: a figure written cleanly and then misread is as worthless as one written wrongly.

What belongs in the tender

1. Is the size of an incoming message checked on the bytes received, and are there upper bounds on its structure?
2. How does the platform recognise a message that is delivered twice?
3. Is the counter increased in a single atomic write, or is it read first?
4. What happens to a write that fails - how often is it retried, at what intervals, and where does it wait in between?
5. Are derived analyses produced only after the count has been secured?
6. At what resolution is data stored, and who decides the resolution an analysis uses?
7. Is planning the counting lines part of the installation, so that two sensors on the same door do not overlap?

How we do it

The ANALYSIT Counting System receives the sensor's live data push over HTTPS at an address belonging to the site, with a rate limit, a bearer key per site compared in constant time, a hard limit of 1 MB on the bytes received, and full schema validation with at most 100 frames and 500 events and 500 objects per frame.

Before any count, ACS atomically claims a duplicate key built from tenant, site, sensor serial number and frame number. The count then goes into the minute bucket as a single atomic upsert, and the site's live counter is incremented just as atomically. Both core write paths run through a dead-letter queue with up to three retries, exponential backoff and jitter; claims that have been in progress for more than five minutes are released automatically.

One document per minute gives 1,440 buckets per site and day, kept for 730 days, with granularity enforced on the server from one minute up to one day. Zone dwell time, journeys, queue situations, alert checks and display triggers are produced only afterwards, from the same push.

If you want to know how your figures come about before they appear in a report to the board, talk to us. We are happy to walk through the ten links with your IT team.