Difference between revisions of "Mac OS X Porting - Apple Remote implementation"

Revision as of 12:53, 29 August 2008

Apple Remote Implementation for OpenOffice.org Aqua on Mac OS X

Description

The remote, on the above picture, allows to drive applications in fullscreen mode, for playing presentations, photos slideshows, play music .. and so on, with the latests computers Apple ships, including Intel processors, but not only.

The most known application running with the remote, is Front Row

There is at least one model running PowerPC processor, the iMac G5 iSight, who is shipped with a remote too.

More information : Apple Remote on Wikipedia

Goal

Timeline

Task started early june 2008

Current status: cws apple_remote01 is complete, and the code under review. Will be set as Ready for QA after code cleanup

Remaining: code cleanup (removing debug infos)

No issue known at this time, just some limitations, explained below.

Source Code authors

Initial code who allows the Apple Remote control use, has been written by from Martin Kahr, under the MIT License. This code, who has been put in the new apple_remote module, has been adapted to OpenOffice.org by Eric Bachard, under the same license.

For further informations, please see : Martin website

Important: to make it work with OpenOffice.org , the initial code in the cws has been modified a lot. Please use the original code if you want to see the diffs

Note: the existing code allows to use the Keyspan remote device. If you have such device, please provide us feedback and help us to make it work

Modes (definitions)

The current implementation is described below. If you have a better idea (not too complicated though), please tell us.

1) Mode not presenting

This mode means, windows and frames are displayed, or the current window has been extended in all the screen (using CTRL+shift +J), just hidding the window , borders,the menubar and the dock.

2) Mode Presentation (fullscreen aka kUIModeAllHidden)

This is a windowless mode, aka fullscreen mode, known as the usual mode for presentation.

3) Other mode : Presenter Screen extension

This is not a mode, but more a feature, allowing you to read your notes on the laptop, while the slides are displayed .. etc

Download the extension, install it, and in the diaporama properties, don't forget to choose screen 2 for the presentation screen (else you'll see the presenter screen on the public screen and the presentation on your laptop / screen :-)

4) Detailed behavior (Draft)

Mode not presenting (Normal Mode)

Either :

Or:

Case A : whatever sheet (even empty):

Case B: when the sheet does contain datas (e.g. we suppose here the dimensions of the sheet are 10 lines / 20 columns)

Base document:

TODO

Draw document:

TODO

Mode Presentation

• Hit Page backward ( |<< symbol ) or - button ( Volume down) gives Previous page (if existing)
• Hit Page forward ( >>| symbol) or + button (Volume up) gives Next page (if existing)
• longHit Page backward ( |<< symbol ) gives First page
• long Hit Page forward ( >>| symbol) gives Last page
• Hold Menu ends the presentation

Not yet implemented :

• Play/Pause (|> || symbol) play/pause media or automation
• hit Menu gives contextual menu ( needs reverse engineering)

Analysis: using Front Row behavior as starting point.


Some definitions: 

A) Normal  mode: 

Either :
- one Impress window is in front (active window)

-> hit play button gives the focus to the remote, and Impress can be started

Or:

- another OpenOffice.org application is focused (is the active Window) : 

Writer, or Calc or Draw, or Base document is focused and is in NOT in fullscreen :

-> hit play button simulates the F5 key, so it will open/close the navigator, if available.

Writer document is focused and is in fullscreen (using CMD+shift+J) :

CMD+shift+J gives a "fullscreen like" document, and then the other keys are available:

-> hit left/right or -/+ key moves the cursor in the text

-> hit long left/ long right or long -/ long + moves the cursor to the begining/end of the line

Calc Document is focused and is in fullscreen (using CMD+shift+J):

* Whatever sheet (even empty):

-> hit left/right or -/+ key moves the cursor in the sheet, whatever the sheet does contain (works with empty sheets)


* When the sheet does contain datas (e.g. we suppose here the dimensions of the sheet are 10 lines / 20 columns)

-> hit long left moves the cursor to the begining of the line, means 1st column of the current line.

-> hit long - does nothing

-> hit long right moves the cursor to the 20th column of the current line

-> hit long + moves the cursor on the 1st line (top of the sheet) of the current column



Base document:

TODO

Draw document:

TODO


B) pause: nothing happens, the displayed slide is infinitely displayed, sound (if ever) is stopped.

C) short key stroke: short key hit, inferior to 0,4 second

D) long key stroke: obtained when the key is maintained hit more than 0,4 second 



E) Presentation mode: 

Starting point: fullscreen, slide 1 is displayed, presentation paused (default)

From presentation mode, can be reached any slide, using the menu (used as a right click)


Expected behavior :

[Menu Key] toggle menu key : 

( later: 1) from IMPRESS normal mode : play long proposes a list of presentations (like Front Row does) )

If and only if Impress window is in front (is the active window) => put the active default selected document in paused presentation mode, aka "Fullscreen" mode.

To return in Impress mode :

Either (if in pause) another hit 
or a long stroke returns ends the fullscreen mode, and returns in Impress ( in windowed mode)


NOT YET IMPLEMENTED 2) equivalent to right click -> gives the contextual menu in presentation mode


[Play]

play : toggle start/pause for the presentation 
play long : toggle presentation mode / normal mode 

[Right] [Left]

In list mode or in menu mode: right left allows to browse the list

presentation mode only:
- normal stroke: right / left :  next slide / previous slide
- long stroke: right/left : got to previous slide/ to next slide *continuously*

Note: go to the first/last slide can be achived using the contextual menu


[+] [-]

Long stroke does not work
FIXME : how retrieve the normal behavior ? Only short one leads to modify the sound level ( what else ? )

starting designing the final binding

Make it buildable

Done. With the current code, no warnings, everything is delivered as expected.

- in apple_remote module (build triggered in postprocess)

- in vcl : apple_remote module has been added as a dependancy

- in scp2 ( libAppleRemotemx{i | p}.dylib has been added in the package

Integrate the code in OpenOffice.org for Mac OS X

Done :

Concerned modules :

• external (Apple Remote code will be there due to the MIT License )
• scp2 : libAppleRemotemxi.dylib has to be packaged
• vcl : MainControler object and notifications are detected by the NSApplication in all its life + add apple_remote module as dependancy
• postprocess : build apple_remote module

Code :

• created external/apple_remote tree. To check out the module (until it is integrated) : cvs co apple_remote
• created the makefile, links against IOKit framework
• modified prj/build.lst to build external/AppleRemote
• modified prj/d.lst to deliver the headers in the solver (vcl will need them) and deliver libAppleRemotemxi.dylib
• build is ok : no warning (fixed the static issue, and all the other warnings , made the code more robust)
• in vcl : modified the makefile ( linking against libAppleRemotemxi.dylib ), added mpMainController member in SalData ( saldata.hxx ),

initialized mpMainController in the SalData Ctor, added the MainController initialization in initNSApp() (salinst.cxx)

• Packaging works as expected, and OpenOffice.org works fine with the new lib included in the archive

Done :

Remote Control initialized, and reports finely all events.

Global Keyboard works

Multi click should work too

Current logs :

Reading symbols for shared libraries . done
System Version 1049
Stored System Version 1049
2008-08-13 11:27:49.513 soffice.bin[13926] RemoteControl initWithDelegate ok
2008-08-13 11:27:49.513 soffice.bin[13926] RemoteControlContainer initWithDelegate ok
2008-08-13 11:27:49.514 soffice.bin[13926] RemoteControl initWithDelegate ok
2008-08-13 11:27:49.516 soffice.bin[13926] [container instantiateAndAddRemoteControlDeviceWithClass: [AppleRemote class]] successfull
2008-08-13 11:27:49.517 soffice.bin[13926] RemoteControlContainer instantiateAndAddRemoteControlDeviceWithClass failed
2008-08-13 11:27:49.518 soffice.bin[13926] [container instantiateAndAddRemoteControlDeviceWithClass: [KeyspanFrontRowControl class]] failed
2008-08-13 11:27:49.518 soffice.bin[13926] RemoteControl initWithDelegate ok
2008-08-13 11:27:49.519 soffice.bin[13926] [container instantiateAndAddRemoteControlDeviceWithClass: [GlobalKeyboardDevice class]] successfull
2008-08-13 11:27:49.519 soffice.bin[13926] MainController init done

Important:

The remote is active when one OpenOffice.org window has the focus. Using the Finder ( Apple + TAB ) deactivates it, and then Front Row becomes again available. The OpenOffice.org application no longer receives the events when Front Row is active.

Select another application gives :

2008-08-13 11:30:00.241 soffice.bin[13927] stopListening to events... 
2008-08-13 11:30:00.242 soffice.bin[13927] key = RemoteControlDeviceName 
2008-08-13 11:30:00.242 soffice.bin[13927] key = CFBundleIdentifier 
2008-08-13 11:30:00.242 soffice.bin[13927] value = AppleIRController 
2008-08-13 11:30:00.242 soffice.bin[13927] value = org.openoffice.script 
2008-08-13 11:30:00.242 soffice.bin[13927] sendDistributedNotification ...
2008-08-13 11:30:00.242 soffice.bin[13927] Notification posted...
2008-08-13 11:30:00.242 soffice.bin[13927] sendFinishedNotifcationForAppIdentifier ...

Note: the keys are extracted from the userInfo dictionary

Select again OpenOffice.org in front gives:

2008-08-13 11:32:23.231 soffice.bin[13927] startListening to events... 
2008-08-13 11:32:23.239 soffice.bin[13927] reset... (after listening to remote)
2008-08-13 11:32:23.239 soffice.bin[13927] Apple Remote will become active - Using remote controls

Current work in progress :

• alias for apple_remote in external created by Martin Hollmichel (see issue 92739 )
• code commited
• cws appleremote01 created ( Florian Heckl will QA it)
• all known issues fixed. Remains: better keycode mapping
• final design for final implementation in progress

Needed:

• infos about contextual menu emulation
• feedback for the Keyspan remote
• feedback for the current Design

Intercept events with the remote, and trace

Adding some NSLog at the right place, we have the numerical values returned by the remote:

• + as value "2"
• - is seen as value "4"
• Menu button is seen as the value "8"
• |> || (play pause) as value "16"
• >>| is seen as value "32"
• <<| is seen as "64"
• long hold with key Menu (kRemoteButtonMenu_Hold) is seen as value "512"
• long hold with key Play (kRemoteButtonPlay_Hold) is seen as value "1024"
• long hold with key Backward (kRemoteButtonLeft_Hold) is seen as value "2048"
• long hold with key Forward (kRemoteButtonRight_Hold) is seen as value "4096"

Important: when in Front Row mode, the events are seen too, and accordingly to the link below, there is a way to programaticaly enable / disable it.

Do not work :

• long hold with key Plus (kRemoteButtonPlus_Hold) is seen as value "128"
• long hold with key Minus (kRemoteButtonMinus_Hold) is seen as value "256"

Added kRemoteButtonNone, for button initialization

Started :

vcl bindin using ImplHandleAppCommand()

Missing (to be added ?) :

// Missing : case kRemoteButtonMenu:

case kRemoteButtonMenu_Hold:

case kRemoteButtonPlay_Hold:

case kRemoteButtonRight_Hold:

case kRemoteButtonLeft_Hold:

case kRemoteControl_Switched:

break;

Done :

case kRemoteButtonPlay

case kRemoteButtonLeft

case kRemoteButtonRight

case kRemoteButtonMinus

case kRemoteButtonPlus

TODO :

• bind with vcl events
• make it work
• improve
• make it work also with OOo Presenter extension

XSlideShowController use

Will be used for the second (more complete) implementation:

• The plan is to use VclSimpleEvent* . For further information, please read

Philipp Lohmann presented me Andre Fisher, the specialist of the thing. Andre kindly explained me where start :

From Andre mail :

First you need access to the XSlideShowController.  You get that from a model (document) like this

    Reference<XModel> xModel; // Given.

    Reference<XPresentationSupplier> xPS (
        xModel, UNO_QUERY_THROW);
    Reference<XPresentation2> xP (
        xPS->getPresentation(), UNO_QUERY_THROW);
    Reference<XSlideShowController> xSSC (
        xP->getController());


The XSlideShowController provides the functionality to change slides and get information about the current slide.

You can find the interfaces in com/sun/star/presentation, and the real implementation in sd/source/ui/slideshow/slideshowimpl.cxx

Usefull variables sin the interface ( XSlideShowController.idl )

boolean isRunning() -> are we in presentation mode, or not ? 

long getSlideCount(): returns the number of slides ( are we at the beggining, the end .. ) 

void gotoNextSlide()  -> kRemoteButtonRight 

void gotoPreviousSlide() -> kRemoteButtonLeft

void pause()  // needs some work

void activate() : activates the user interface of the slideshow -> can we use the FrontRow menu ? 

boolean isEndless() : demo, the presentation is running in infinite loop when true

bind with vcl events

=> Work in progress (new events added in the experimental changes)

First basic implementation works :

• play
• quit presentation mode
• next slide
• previous slide
• first slide
• last slide
• up : volume up does not work with the current implementation, but is replaced with previous slide
• down: volume down does not work with the current implementation, but is replaced with next slide

Missing :

• contextual menus

Make it work

=> DONE TODO

Create the cws and commit

TODO: file the issue, and ask mh the alias to be created. (maybe things are more easy now ?)

Improve

• Did Apple document the API since ?
• Document how things are working '[started]
• Does an Impress Controller API exist ?
• Find where the events are managed in slideshow (or sd) ? [ask Thorsten]
• Contact Andre Fisher, the specialist of the thing (following Philipp Lohmann recommandations), and see how marry the remote with the presenter screen

Code description

Files :

TODO DONE

Using different devices

Right now the wrapper ships with support for three devices:

• [works] AppleRemote: Apple Remote Control
• [works] GlobalKeyboardDevice: Registers global keyboard shortcuts to provide a virtual remote control
• [untested, no feedback yet] KeyspanFrontRowControl: Keyspan RF Remote for FrontRow

Links

iRemoted

HID at usb.org

Using the Apple Remote Control