Mac OS X Porting - Apple Remote implementation
Contents
- 1 Apple Remote Implementation for OpenOffice.org Aqua on Mac OS X
- 2 4) Detailed behavior after appleremote02 will be integrated (Draft)
- 3 Technical part
- 4 Intercept events with the remote, and trace
- 5 XSlideShowController use
- 6 bind with vcl events
- 7 Make it work
- 8 Create the cws and commit
- 9 Links
- 10 People involved
Apple Remote Implementation for OpenOffice.org Aqua on Mac OS X
Description
The remote, on the above picture, allows to drive applications in fullscreen mode, for playing presentations, photos slideshows, play music .. and so on, with the latests computers Apple ships, including Intel processors, but not only.
The most known application running with the remote, is Front Row
There is at least one model running PowerPC processor, the iMac G5 iSight, who is shipped with a remote too.
More information : Apple Remote on Wikipedia
Goal
Several steps are scheduled (appleremote01, appleremote02 and appleremote03)
Timeline
Task started early june 2008
CWS name | State | Date | milestone | comment | Owner | QA resp. | |
---|---|---|---|---|---|---|---|
appleremote01 | ready for QA | 21st September 2008 | ericb | Florian Heckl | done | ||
appleremote01 | integrated | 27th October 2008 | DEV300_m35 | Apple Remote will be in 3.1 | ericb | Florian Heckl | done |
appleremote02 | created | 28th October 2008 | DEV300_m37 | replace keyCodes, using MEDIA_COMMAND_* instead |
ericb | done | |
appleremote02 | Ready for QA | scheduled : mid January 2009 | planned : DEV300_m39 | ericb | Wolfram Garten | done | |
appleremote02 | Integrated | scheduled : January 2009 (should be in 3.1) |
cws approved by QA (14th January) planned : DEV300_m39 |
ericb | Wolfram Garten | ||
appleremote03 | planned | 14th December 2008 | Try to use the contextual menu in presentation mode (preliminary step) |
ericb | done | ||
appleremote03 | created | 14th January 2009 | DEV300_m38 | ericb | |||
appleremote03 | Ready for QA | undefined | planned : DEV300_mxx | ericb |
CWS3: appleremote03
The plan is :
- trace what happens when using contextual menus
- understand what can be used for the control
- propose a solution (if possible)
- implement the list mode
- test
- verify
- integrate
TODO :
- activate the menus
- define the expected behavior
- implement the missing callbacks
- ... (fixme)
DONE January 14th 2009
- appleremote03, based on DEV300_m38 (DEV300_m39 not ready) being created
December 14th 2008
* Learn how works the Cocoa menus, and see what can be done with contextual menus.
CWS2: appleremote02
Next steps:
- integrate appleremote02 changes in time for 3.1
January 14th 2009
- cws Approved by QA
January 12th 2009
- cws marked as Ready for QA
- started a clean build, based on appleremote02
January 11th 2009
- all needed changes commited in appleremote02
January 10th 2009
- updated the wiki
- set target 3.1 for all issues associated to the cws
- now, bOnce is set to true only after addEventListener is executed
- Fixed issue 97925
January 9th 2009
- implementing the addEventlistener in SdDLL::Init() using the static SdDLL::Create() (proposed by Andre Fischer)
- simplified the code : now Windowed mode works, including the Play/pause
- did code cleanup (remove some debug extra stuff e.g.)
- added singleton in sdmod1.cxx, to be sure only one eventlistener has been added
- fixed MEDIA_COMMAND_VOLUME UP/DOWN in slideshowimpl.cxx ( only QUARTZ )
- added a removeEventlistener (in SdModule dtor))
December 19th 2008
- new event listener added in sd (FIXME: maybe not the best location)
- new implementation works (including the F5 button)
- everything works : cws completed (if ok with the event listener)
- tests in progress (needs some profiling, and maybe a removeEventlistener somewhere)
- Next step : make it ready for QA
December 13rd 2008
- new implementation works (excepted the F5 button)
- PLAY/PAUSE now works (blanking the screen) if hit during the slideshow
- contextual menus have been postponed to appleremote03 : it works, but there is currently no way to control what the Cocoa menu do (everything is stalled in OOo when the menus are open) -> appleremote03
- the F5 button will work once the slideshow will register as event listener more early : work in progress (waiting for issue 97195 to be solved]
CWS1: appleremote01
Current status:
- cws appleremote01 integrated
- cws appleremote01 approved by QA and nominated
- all debug informations have been removed: the apple remote now works well, but silently if DEBUG is not defined at buildtime
- appleremote01 cws is sync'ed with DEV300_m31
- the changes Florian requested have been commited (added a release for mpMainController in Saldata Dtor, put headers in apple_remote/inc ... )
- cws appleremote01 is complete,
- Florian Heckl (as QA resp. ) reviewed the code,
Plan
Once appleremote01 will be integrated, a second cws, namely appleremote02 will be created, and will implement the missing features, described below, and not included in appleremote01
The plan is to use another type of events : NSApplicationDefined , and pass the buttonIdentifier as parameter (data1 )
More technicaly, I created a new events subtype ( AppleRemoteControlEvent), and this new events type will be detected by the NSApp in AquaSalInstance::handleAppDefinedEvent( NSEvent* pEvent ).
I didn't change anything to the buttons, but I'll probably add new effects, like contextual menus, once they will be properly handled.
First results :
- in Fullscreen
Received the following event from the Apple Remote : 16
[... cut other values ... ]
Received the following event from the Apple Remote : 512
Received the following event from the Apple Remote : 2
Received the following event from the Apple Remote : 4
Received the following event from the Apple Remote : 2048
Received the following event from the Apple Remote : 4096
- in Normal mode
Received the following event from the Apple Remote : 16
Received the following event from the Apple Remote : 512
-> ok !!
To simplify the work, we'll use the existing Windows implementation vcl/source/win/window/salframe.cxx (in static void ImplHandleAppCommand( HWND hWnd, LPARAM lParam ) )
Done (but code not commited, because appleremote01 is not integrated )
- [DONE] remove the event sending the keycode
- [DONE] implement new method, sending new event type, including the buttonIdentifier value
- [DONE] implement glu code for intercepting the event and receiving the value of the message
- [DONE] new libAppleRemote builds without any glitch
- [DONE] turn the notification into the right constant (RemoteControl.h does contain the values, we'll bind with the one defined in vcl/inc/vcl/cmdevt.hxx )
- [DONE] build (should be not too much of problems)
- [DONE] go to the first slide now works
- [DONE] go to the last slide now works
- [DONE] : the Apple Remote works with the new implementation !!
- [DONE] contextual menu works, but does nothing : FIXME : pl explained me this is a problematic part, because Cocoa does take over everything, and OOo has no
- [DONE] commit: waiting for DEV300_m35, since appleremote01 will be integrated in this milestone :-)
- [DONE] fix: start not working because not implemented in the slideshow
- [DONE] commit the changes in appleremote02 cws
access until the popup menu closes.
Planned :
- wait for confirmation the code is ok (better stay prudent)
- provide a build for testing purpose
- test
- verify
Source Code authors
Initial code who allows the Apple Remote control use, has been written by from Martin Kahr, under the MIT License. This code, who has been put in the new apple_remote module, has been adapted to OpenOffice.org by Eric Bachard, under the same license.
For further informations, please see : Martin Khar website
Important: to make it work with OpenOffice.org , the initial code in the cws has been modified a lot. Please use the original code if you want to see the diffs
Note: the existing code allows to use the Keyspan remote device. If you have such device, please provide us feedback and help us to make it work
Modes (definitions)
The current implementation is described below. If you have a better idea (not too complicated though), please tell us.
1) Mode not presenting
This mode means, windows and frames are displayed.
If you have choosen to run the prosentation in Windowed mode, the remote works like if you are in fullscreen (since appleremote02)
Else, just play will work, and start the presentation in fullscreen.
2) Mode Presentation
Starting point: fullscreen, slide 1 is displayed, presentation paused (default). From presentation mode, can be reached any slide, using the menu (used as a right click)
Or :
The current window has been extended in all the screen (using CTRL+shift +J), just hidding the window , borders,the menubar and the dock.
This is a window-less mode, aka fullscreen mode, known as the usual mode for presentation.
3) LIST_MODE (will be implemented in appleremote03)
=> DRAFT
What follows is just a draft: means there is no guarantee it will work as expected, and will need a lot of tests.
Expected behavior: in list mode or in menu mode: [right], [left], [+] or [-] buttons allow to browse the contextual menu
Important : this feature works in parallel with other events from mouse or keyboard
Hit menu when on fullscreen (during the presentation) :
- pauses the presentation
- makes the contextual menu appear
Hit the menu entry again or escape :
- removes the contextual menu
- resumes, and the presentation continues
To navigate :
Hit the - key goes down in the menu when possible
Hit the +key goes up in the menu when possible
Hit the right (forward) key goes right in the menu when possible
Hit the left (backward) key goes left in the menu when possible
4) Other mode : Presenter Screen extension
This is not a mode, but more a feature, allowing you to read your notes on the laptop, while the slides are displayed .. etc
Download the extension, install it, and in the diaporama properties, don't forget to choose screen 2 for the presentation screen (else you'll see the presenter screen on the public screen and the presentation on your laptop / screen :-)
Other definitions
short key stroke: short key hit, inferior to 0,4 second
long key stroke: obtained when the key is maintained hit more than 0,4 second
4) Detailed behavior after appleremote02 will be integrated (Draft)
Mode not presenting (aka Windowed mode)
Either :
and if the Windowed mode is selected in the preferences :
and every button of the remote work in Windowed mode (excepted the play/pause button).
Hold Play button quits the presentation
else :
Play only will start the presentation at the current slide (common use)
Or:
Or:
Or:
Base document:
TODO
Draw document:
TODO
Mode Presentation
- Hit Page backward ( |<< symbol ) or + button ( Volume down) gives Previous page (if existing)
- Hit Page forward ( >>| symbol) or - button (Volume up) gives Next page (if existing)
- longHit Page backward ( |<< symbol ) gives First page
- long Hit Page forward ( >>| symbol) gives Last page
- Hold Menu ends the presentation
- Play : toggle play/pause button can play pause automatic presentation
Other PRESENTATION_MODE FEATURES
1) + / - button can increase / decrease sound volume (very unsure, because some Apple black magic seems to lock that buttons)
2) Hit menu key simulates right click -> gives the contextual menu in presentation mode => opens the LIST MODE
3) hold play (long hit) allows to choose a list of presentations, in some default dir (like Front Row does) )
4) in LIST_MODE (available in presentation mode only)
- normal stroke: [right], [left], [+] or [-] browses the contextual menu
- play validates the choice ( ends the LIST_MODE )
- long stroke menu : simulates escape and ends the LIST_MODE
IMPORTANT: long stroke menu in LIST_MODE has not the same effect than the long stroke menu when NOY in LIST_MODE
Note: go to the first/last slide can be achived using the contextual menu
Technical part
Make it buildable
Done. With the current code, no warnings, everything is delivered as expected.
- in apple_remote module (build triggered in postprocess)
- in vcl : apple_remote module has been added as a dependancy
- in scp2 ( libAppleRemotemx{i | p}.dylib has been added in the package
Integrate the code in OpenOffice.org for Mac OS X
Done :
Concerned modules :
- external (Apple Remote code will be there due to the MIT License )
- scp2 : libAppleRemotemxi.dylib has to be packaged
- vcl : MainControler object and notifications are detected by the NSApplication in all its life + add apple_remote module as dependancy
- postprocess : build apple_remote module
Code :
- created external/apple_remote tree. To check out the module (until it is integrated) : cvs co apple_remote
- created the makefile, links against IOKit framework
- modified prj/build.lst to build external/AppleRemote
- modified prj/d.lst to deliver the headers in the solver (vcl will need them) and deliver libAppleRemotemxi.dylib
- build is ok : no warning (fixed the static issue, and all the other warnings , made the code more robust)
- in vcl : modified the makefile ( linking against libAppleRemotemxi.dylib ), added mpMainController member in SalData ( saldata.hxx ),
initialized mpMainController in the SalData Ctor, added the MainController initialization in initNSApp() (salinst.cxx)
- release mpMainController in saltada destructor
- Packaging works as expected, and OpenOffice.org works fine with the new lib included in the archive
Done :
Remote Control initialized, and reports finely all events.
Global Keyboard works
Multi click should work too
Current logs :
Reading symbols for shared libraries . done System Version 1049 Stored System Version 1049 2008-08-13 11:27:49.513 soffice.bin[13926] RemoteControl initWithDelegate ok 2008-08-13 11:27:49.513 soffice.bin[13926] RemoteControlContainer initWithDelegate ok 2008-08-13 11:27:49.514 soffice.bin[13926] RemoteControl initWithDelegate ok 2008-08-13 11:27:49.516 soffice.bin[13926] [container instantiateAndAddRemoteControlDeviceWithClass: [AppleRemote class]] successfull 2008-08-13 11:27:49.517 soffice.bin[13926] RemoteControlContainer instantiateAndAddRemoteControlDeviceWithClass failed 2008-08-13 11:27:49.518 soffice.bin[13926] [container instantiateAndAddRemoteControlDeviceWithClass: [KeyspanFrontRowControl class]] failed 2008-08-13 11:27:49.518 soffice.bin[13926] RemoteControl initWithDelegate ok 2008-08-13 11:27:49.519 soffice.bin[13926] [container instantiateAndAddRemoteControlDeviceWithClass: [GlobalKeyboardDevice class]] successfull 2008-08-13 11:27:49.519 soffice.bin[13926] MainController init done
Important:
The remote is active when one OpenOffice.org window has the focus. Using the Finder ( Apple + TAB ) deactivates it, and then Front Row becomes again available. The OpenOffice.org application no longer receives the events when Front Row is active.
Select another application gives :
2008-08-13 11:30:00.241 soffice.bin[13927] stopListening to events... 2008-08-13 11:30:00.242 soffice.bin[13927] key = RemoteControlDeviceName 2008-08-13 11:30:00.242 soffice.bin[13927] key = CFBundleIdentifier 2008-08-13 11:30:00.242 soffice.bin[13927] value = AppleIRController 2008-08-13 11:30:00.242 soffice.bin[13927] value = org.openoffice.script 2008-08-13 11:30:00.242 soffice.bin[13927] sendDistributedNotification ... 2008-08-13 11:30:00.242 soffice.bin[13927] Notification posted... 2008-08-13 11:30:00.242 soffice.bin[13927] sendFinishedNotifcationForAppIdentifier ...
Note: the keys are extracted from the userInfo dictionary
Select again OpenOffice.org in front gives:
2008-08-13 11:32:23.231 soffice.bin[13927] startListening to events... 2008-08-13 11:32:23.239 soffice.bin[13927] reset... (after listening to remote) 2008-08-13 11:32:23.239 soffice.bin[13927] Apple Remote will become active - Using remote controls
TODO DONE :
- alias for apple_remote in external created by Martin Hollmichel (see issue 92739 )
- code commited
- cws appleremote01 created ( Florian Heckl will QA it)
- all known issues fixed. Remains: better keycode mapping
- final design for final implementation in progress
Needed:
- infos about contextual menu emulation
- feedback for the Keyspan remote
- feedback for the current Design
Intercept events with the remote, and trace
Adding some NSLog at the right place, we have the numerical values returned by the remote:
- + as value "2"
- - is seen as value "4"
- Menu button is seen as the value "8"
- |> || (play pause) as value "16"
- >>| is seen as value "32"
- <<| is seen as "64"
- long hold with key Menu (kRemoteButtonMenu_Hold) is seen as value "512"
- long hold with key Play (kRemoteButtonPlay_Hold) is seen as value "1024"
- long hold with key Backward (kRemoteButtonLeft_Hold) is seen as value "2048"
- long hold with key Forward (kRemoteButtonRight_Hold) is seen as value "4096"
Important: when in Front Row mode, the events are seen too, and accordingly to the link below, there is a way to programaticaly enable / disable it.
Do not work :
- long hold with key Plus (kRemoteButtonPlus_Hold) is seen as value "128"
- long hold with key Minus (kRemoteButtonMinus_Hold) is seen as value "256"
Added kRemoteButtonNone, for button initialization
Started :
vcl bindin using ImplHandleAppCommand()
Missing (to be added ?) :
// Missing : case kRemoteButtonMenu:
case kRemoteButtonMenu_Hold:
case kRemoteButtonPlay_Hold:
case kRemoteButtonRight_Hold:
case kRemoteButtonLeft_Hold:
case kRemoteControl_Switched:
break;
Done :
case kRemoteButtonPlay
case kRemoteButtonLeft
case kRemoteButtonRight
case kRemoteButtonMinus
case kRemoteButtonPlus
TODO :
- bind with vcl events
- make it work
- improve
- make it work also with OOo Presenter extension
XSlideShowController use
- Since another solution has been used, what follows it there for information, as developer note
Philipp Lohmann presented me Andre Fisher, the specialist of the thing. Andre kindly explained me where start :
From Andre mail :
First you need access to the XSlideShowController. You get that from a model (document) like this Reference<XModel> xModel; // Given. Reference<XPresentationSupplier> xPS ( xModel, UNO_QUERY_THROW); Reference<XPresentation2> xP ( xPS->getPresentation(), UNO_QUERY_THROW); Reference<XSlideShowController> xSSC ( xP->getController()); The XSlideShowController provides the functionality to change slides and get information about the current slide. You can find the interfaces in com/sun/star/presentation, and the real implementation in sd/source/ui/slideshow/slideshowimpl.cxx Usefull variables sin the interface ( XSlideShowController.idl ) boolean isRunning() -> are we in presentation mode, or not ? long getSlideCount(): returns the number of slides ( are we at the beggining, the end .. ) void gotoNextSlide() -> kRemoteButtonRight void gotoPreviousSlide() -> kRemoteButtonLeft void pause() // needs some work void activate() : activates the user interface of the slideshow -> can we use the FrontRow menu ? boolean isEndless() : demo, the presentation is running in infinite loop when true
bind with vcl events
=> DONE TODO
First basic implementation works :
- play
- quit presentation mode
- next slide
- previous slide
- first slide
- last slide
- up : volume up does not work with the current implementation, but is replaced with previous slide
- down: volume down does not work with the current implementation, but is replaced with next slide
Missing :
- contextual menus
Make it work
=> DONE TODO
Create the cws and commit
DONE: file the issue, and ask mh the alias to be created. (maybe things are more easy now ?)
Improve
- Did Apple document the API since ?
- Document how things are working '[started]
- Does an Impress Controller API exist ?
- Find where the events are managed in slideshow (or sd) ? [ask Thorsten]
- Contact Andre Fisher, the specialist of the thing (following Philipp Lohmann recommandations), and see how marry the remote with the presenter screen
Code description
Files :
TODO DONE
Using different devices
Right now the wrapper ships with support for three devices:
- [works] AppleRemote: Apple Remote Control
- [works] GlobalKeyboardDevice: Registers global keyboard shortcuts to provide a virtual remote control
- [untested, no feedback yet] KeyspanFrontRowControl: Keyspan RF Remote for FrontRow
Links
Using the Apple Remote Control
People involved
Name | OOo Nickname | Role |
---|---|---|
Eric Bachard | ericb2 | Development |
Philipp Lohmann | pl | Code review (vcl, apple_remote) |
Christian Lippka | cl | Code review (sd, MEDIA_COMMAND ) |
Andre Fischer | af | Code review (sd) |
Florian Heckl | fheckl | QA appleremote01 |
xxxx | xxxx | User Experience |
Eric Bachard | ericb2 | Documentation |