Vision Pro Connectivity
One thing everyone agrees on is that using the Vision Pro is a solitary experience. Apple has done plenty to make the experience more inclusive, including Personas, SharePlay, Immersion and EyeSight. The issue is still people around you can’t see what you’re doing on the Vision Pro unless they also have a device and you can co-habit the space and each see what the other sees. But let’s be realistic, most households and even businesses are likely to only have one of these to share around, at least for the time being.
I have been testing out how it might be possible to connect the Vision Pro to other devices, particularly iPhone’s and the Apple Watch, although the concepts here also apply to all of Apple’s ecosystem of products.
Connectivity
It used to be the case that a bonjour connection was required to create a local network and communicate between devices, or it’d be necessary to create a Bluetooth channel between devices. Although these are still the case, Apple now provides frameworks to take away much of the low level, including Multipeer Connectivity and Watch Connectivity. Unfortunately neither of these works between all three devices, the iPhone and Apple Watch support Watch Connectivity, the Vision Pro and the iPhone support Multipeer Connectivity, so there’s a little juggling involved to get all three devices talking to each other, but it’s still possible.
In my proof of concept application I have a 3D scene on the Vision Pro that reacts to instructions sent by the iPhone and a connected Apple Watch (via the phone). Although the messages are simple strings in this case, anything that can conforms to Codable is fair game and in my experiments the speed is more than adequate for most use cases. After all this technology is what powers many of Apple's multi device features so has been optimised for realtime communication.
Setup
There’s no real different in the setup here than what’s already documented for tapping into the connectivity frameworks on other devices. The only gotchas are that the helper frameworks such as MCBrowserViewController aren't available on the Vision Pro, we have to do the leg work ourselves and even though the Vision Pro has the framework for Watch Connectivity, the availability for connections is false (as of VisionOS 1.1) so isn’t (yet) a possibility here.
To get it up and running on the Vision Pro, we setup a session, and broadcast ourselves as a host, as well as listen for any communication on that channel.
class PeerConnectivity: NSObject {
var peerID: MCPeerID
var mcSession: MCSession
var advertiser: MCNearbyServiceAdvertiser
var browser: MCNearbyServiceBrowser
var messageDelegate: MessageDelegate?
let serviceType = "interapp"
override init() {
peerID = MCPeerID(displayName: UIDevice.current.name)
mcSession = MCSession(peer: peerID)
advertiser = MCNearbyServiceAdvertiser(peer: peerID, discoveryInfo: nil, serviceType: serviceType)
browser = MCNearbyServiceBrowser(peer: peerID, serviceType: serviceType)
super.init()
mcSession.delegate = self
advertiser.delegate = self
browser.delegate = self
}
func startHosting() {
advertiser.startAdvertisingPeer()
}
}On the iPhone it’s a bit more complicated as there’s both a Multipeer and Watch Connectivity to contend with. First we start a Multipeer session and for simplicity auto join any available hosts. We also create a Watch Connectivity session and make sure to listen out for any messages on there too. These messages are forwarded onto any available Multipeer connection, which works surprisingly well speed and reliability wise in testing.
//received a Watch connectivity Message, process and forward onto Peer Connectivity
func gotMessage(message: String) {
print("got message ConnectionModel: \(message)")
self.sharedValue = message
self.commands.append(message)
print("commands: \(commands.count)")
//just send it directly
peerConnection?.sendBallCommand(command: message)
}
func sendBallCommand(command: String) {
let message = ["ball": command]
peerConnectivity.sendMessage(message)
}
func sendMessage(_ message: [String: Any]) {
if mcSession.connectedPeers.isEmpty {
print("no peers")
return
}
do {
guard JSONSerialization.isValidJSONObject(message) else {
print("message not validJSON: \(message)")
return
}
let data = try JSONSerialization.data(withJSONObject: message, options: .prettyPrinted)
try mcSession.send(data, toPeers: mcSession.connectedPeers, with: .reliable)
} catch {
print("error sending: \(error)")
}
}The Watch is all about Watch Connectivity, which in theory is simple, but there’s a few gotchas involved in recreating the session in certain circumstances, these are well documented, but it can be easy to miss edge cases with so much of the underlying connection abstracted away it's also hard to decipher problems when the do go wrong.
class MyWatchConnectivity: NSObject, WCSessionDelegate {
private let session: WCSession
var messageDelegate: MessageDelegate?
init(session: WCSession = .default) {
self.session = session
super.init()
session.delegate = self
connect()
}
func connect() {
if WCSession.isSupported() {
print("activating session")
session.activate()
}
}
func send(message: [String:Any]) -> Void {
session.sendMessage(message, replyHandler: nil) { error in
print("error send: \(error)")
}
}
}That's it, all very simple. I've confirmed this also works on a real device (one of my initial worries).
Final Thoughts
My hope is that this kind of technique will lead to some interesting ideas in the way in which people can be connected through either games or experiences when not everyone has a Vision Pro or even if as an individual you want to send data between devices. Of course this isn’t limited to messages sent to the Vision Pro, there’s no reason it can’t send updates to its state to other devices, why not share what the Vision Pro user is doing to the outside world, without having to screen share.
Some ideas I’ve toyed with include:
Using the Apple Pencil for more accurate input.
Haptics on the Watch to reflect actions on the Vision Pro and advanced "gestures" available from a Watch App.
A game where the input from an opponent’s phone requires a reaction from the Vision Pro wearing user.
I look forward to seeing what more fun things can be achieved, do let me know if you come up with anything exciting.