A camera on macOS, captured through ffmpeg's AVFoundation input.
Declared like any other sensor:
topology do
link :head do
sensor :eye, {BB.Camera.AVF, device: "FaceTime HD Camera", width: 1280, height: 720}
end
endConsumers never talk to this module. They go through BB.Camera, which
refcounts them and asks the driver to start capturing when the first one
arrives and to stop when the last one leaves. Two browser tabs on the same
camera produce one ffmpeg, not two.
What it captures
JPEG, encoded by ffmpeg. AVFoundation hands ffmpeg decoded buffers even from
a USB webcam that sends MJPEG, so there is no passthrough path: every frame
costs an encode. :input_format chooses which pixel format AVFoundation is
asked for, and :quality the encoder's trade between size and fidelity.
Camera access
macOS grants camera access to an app, and a process started from a terminal inherits the terminal's grant. The first capture from a new terminal app raises the system prompt. A VM started by something with no user interface - a launchd job, a daemon, an agent runner - may get neither a prompt nor frames: ffmpeg opens the camera and then waits forever.
So capture gives up when no frame arrives within :start_timeout_ms, kills
ffmpeg, and reports {:no_frames, ms} as the reason in
BB.Error.Hardware.Camera.CaptureFailed. Nothing can tell that case apart
from a camera that simply sends nothing, so the error names the timeout,
not a cause.
What it reports
Capture is confirmed to the hub on the first whole image, not when ffmpeg
starts. Port.open/2 returning says only that ffmpeg exists — a missing
camera would otherwise announce BeginCapture a moment before failing.
Dimensions come from the SOF marker of each frame, which is what was
actually encoded. They are not always the numbers in the DSL: when
AVFoundation refuses ffmpeg's configuration, ffmpeg warns and captures in
the camera's default format instead, and BB.Camera.stream_info/2 reports
the real size.
The frame rate in stream_info is the one that was requested, not the
negotiated one. This driver does not query the device's rate back and does
not parse it out of ffmpeg's console output, so claiming it as actual would
be a guess dressed up as a measurement. What the frames actually did is
measured by BB.Camera.Hub, which sees every one of them, and reported
separately in BB.Message.Camera.CaptureState.
Nothing here enumerates the camera's capabilities. Ask for a size or rate the camera does not list and ffmpeg exits, printing every mode the camera does list on the console - which is also the quickest way to find them out.
Frames do not go on pubsub
A robot's pubsub carries capture lifecycle only:
BB.Message.Camera.BeginCapture and EndCapture, published by the hub.
Frame bytes travel over BB.Camera's bounded media interface, and never
appear in an event or an error.
Options
See options_schema/0. The ones most worth knowing about are :device,
:width, :height and :frame_rate; the rest are bounds with defaults that
suit a desktop and may want lowering on a small target.
Any of them can be filled from a parameter with param/1, and changing that
parameter while the robot runs reconfigures the camera - see
handle_options/2.
Limitations
- macOS only. It drives AVFoundation through ffmpeg. The ffmpeg-tagged tests run anywhere, against a synthetic source.
- Cameras are chosen by name or index, not by unique ID; see
BB.Camera.AVF.Command. - Timestamps are arrival times. The raw MJPEG stream carries no
timestamps of its own, so a frame is stamped when the read completing it
reached the BEAM. See
BB.Camera.FFmpeg.Capturefor what that does and does not mean. - A failed capture is not retried. The failure is reported and capture ends; reopening the device is the consumer's decision, taken by subscribing again. A driver that quietly reopened hardware on its own would be claiming to capture during a window where it is not.
- Changing an option that reaches ffmpeg restarts it, because ffmpeg configures the camera once, when it opens it. Capture ends and begins again around the change, with the events to match; it is not a seamless transition and does not pretend to be.
- ffmpeg's diagnostics go to the VM's console, not into the error. They cannot be merged into the frame stream without corrupting it.
Summary
Functions
Take up options that changed while the robot was running.
The format this camera is declared to produce.
Functions
Take up options that changed while the robot was running.
Reached when a param/1 reference in the camera's DSL options resolves to a
new value. Only a change that alters the ffmpeg invocation restarts capture:
ffmpeg configures the camera once, when it opens it, so a new size or rate
means a new ffmpeg. Capture stops and starts again around the change; it
is not a seamless transition and does not pretend to be. An option that
doesn't reach the command line - a framer bound, say - takes effect without
disturbing the stream.
New options this driver cannot use stop the camera. handle_options/2
has no way to refuse a change and stay running, so a value that satisfies
the parameter's declared type but not this driver - a width of -1 against
a :pos_integer - takes the sensor down, and down again on each restart,
because the parameter was stored before anyone was asked about it. Declare
parameters with bounds that match the options they feed.
The format this camera is declared to produce.
Pure: it is called during discovery against robots that are not running, so it reads the options and nothing else. Options that still hold unresolved parameter references cannot imply a format, and produce an error rather than a guess.