Improving Large-Group Collaboration (LGC)

Kinopio Design
1

LGC = Large Group Collaboration

Problem

When 6+ people are actively collaborating on a space (LGC), the number of broadcast events being received and processed per frame (especially painting, cursor position) is too much work for client computers, and stresses the kinopio-server which acts as a broadcast relay.

Goals

  • LGC shouldn’t sink client performance
  • LGC shouldn’t cause server outages

Context: How Collaboration Works Today in Kinopio

Each client (eg a kinopio web-browser window/session) opens a websocket connection to the server when it loads a space. When you performs an event on a client, that event is broadcast to the server. The server has a simple broadcast relay system that then sends that event to all other clients connected to the space.

The list of broadcast events includes:

  • painting (sent per frame during painting)
  • cursor position (send per frame while cursor is in viewport)
  • card moving (sent per frame while card is being moved)
  • card is being edited
  • card is edited (eg name changed)
  • new connection is being created
  • connection is being edited (eg change color/show label)

In LGC, sending per frame broadcasts means a lot is being relayed and processed, up to 60fps per client. In a scenario with 6 clients where everyone is moving or painting, that’s potentially 6*60 = 360 broadcasts being received and processed every second. With 30 clients, that jumps to 1800 broadcasts per second. Yikes.

Other Approaches Elsewhere

Trello/Glitch

Trello’s client works like Kinopio’s, but Trello clients only broadcast low resolution events (e.g. instead of showing a card being dragged in real time, trello only shows/broadcasts the new card position after the drag is completed). This is fine for an enterprise kanban situation, but (default) low resolution collaboration is less fun and doesn’t fit Kinopio’s interaction model very well.

Figma/Mural

Probably the closest parallel. I’m not really a figma. user so I have no insight into how it holds up to LGC IRL. They don’t have the technical burden of painting though which probably helps a lot (maybe that’s enough?).

I found https://www.figma.com/blog/how-figmas-multiplayer-technology-works , but it doesn’t explicitly mention about how LGC/high numbers of simultaneous updates are handled. Looks like Figma has a max 50 collaborator limit (https://forum.figma.com/t/are-there-any-collaborator-limits-per-project-file/1150/4)

Mural has a help doc that mentions that perf is expected to slow down during LGC: https://support.mural.co/en/articles/5444182-best-practices-for-better-performance-in-mural

Based on this, I don’t think these work any differently from Kinopio, the main diff is that they don’t have painting which is a relatively intense CPU/GPU operation.

Nodenogg.in

Gives the user configurable modes that lets you only see your cards, which allows the user to elect out of handling broadcast events. The technically easiest solution, but actively having to choose and switch modes for handling technical issues is not in the spirit of a toolbar/mode-less tool like Kinopio.

Fighting games/GGPO/rollback-networking

Largely irrelevant to Kinopio , but kinda neat to learn about. GGPO is mainly for situations that require realtime historical event synchronization and correction, and to situations where possible inputs are limited. Really smart idea that basically works by assuming that the input that will happen next is similar to the input that happened before. If I walked forward in the last frame I’ll probably walk forward in the next. In the case that the prediction is wrong, the game ‘rolls back’ to the correct sequence of events.

Because of its cleverness, the rollback system falls apart really fast with anything other than two clients (eg P1 vs P2).

Proposed Solution(s)

  1. Possibly dynamically adjusting the resolution (timing) of when broadcasts are sent, based on number of clients.

  2. Or, not broadcasting painting events during LGC, and showing a notification to inform the user that painting isn’t shown in LGC (easier to do, but less elegant). I’m not sure how effective just targeting painting alone will be irl yet, but it might be worth doing as an easier to implement experiment.

Other ideas?

(also to do:)

  • Investigate possible memory leak issues in the server broadcast relay where old clients may not be getting cleaned up
1 Like
2

possible thresholds:

5+ collabs = turn off paint broadcasting
15+ collabs = turn off broadcasting cursor position
50+ = turn off editing? (I probably don’t need to worry about this number rn tho)

in both bases the user is informed about the feature that’s being disabled and why. These thresholds, may also have a sidebenefit of reducing visual chaos in LGC situations.

Do these numbers feel right to you?

1 Like
3

I’ve also used Miro for live collaboration. I was just in a session with about 40 people with cursors flying around and it seemed okay/fine. I searched a little on details and found these:

Miro has the option for the client to turn off others’ cursors. Not in the spirit of Kinopio, but still an option. The other tip they had was to limit the number of folks with edit access. In other words, presumably, users with read-only would not broadcast events.

In general, yes. And I know paint broadcasting is expensive, but for collab sessions, it is nice to use painting to get people’s attention and gesture on the space. In lieu of that, I have also dragged around a card (with a picture of my face on it) for this purpose.

1 Like
4

What makes painting unique is that the work isn’t just to display a single point (like it is w cursor position), but also the additional work/calculations to fade out those strokes. I wonder if there’s still efficiency to add to remote painting/fading-out on the client side…

1 Like
5

I hear you re modeless/ toolbar less however I do believe Kinopio would benefit from a small screen quick capture mode, I built the proof of concept with the API but I would have had to add a lot more to make it simple to log in / select your space etc which would require me to effectively re-writing a lot of what is already done just for a single input.

This could be modeless by only working on screen sizes under iPad mini for example. A lot of collaborative work we do is gathering thoughts and a lot of the time students are on phones for this first exercise.

It kinda connects to the performance issue as not everyone would be broadcasting / receiving as many events as then the moving and injecting is also more thoughtful mode of working.

Although of course we had slow down with load of cards on screen anyway but that may have been server leak as well

2 Likes
6

what do you mean by this? capturing a screenshot?

1 Like
7

oh also the issue with the server overloading has now been resolved, I should’ve updated this too

1 Like
8

the main issues left is reducing client browser cpu usage during heavy collab

1 Like
9

No I’m pretty sure @adamprocter means a feature that is hyper-focused on allowing the user to rapidly create cards. A few months ago, Adam prototyped a little web app that does this using the API. We also had a brainstorming session about it.

I think the key here is separating the activity of creating card content from positioning/organizing/connecting it. So, in my mind, another piece of this is a temporary “shelf” or “inbox” where these cards go, and from there you can triage them – place them in a space, or even put them in another space.

1 Like
10

i would like to explore this with a browser integration/plugin/extension in the future

2 Likes
11

yes thats what I meant

2 Likes
12

The Siri Shortcut using the API works great for this.

1 Like