Adding Custom Tools into Maya

 

 

When adding a custom tool into Maya, generally we have to start by providing a Context with which the tool will work. Generally speaking, a context is a camera view within maya. By providing our own context within Maya, we are able to handle user input and render our own overlays for a specific viewport.

When adding any tool into Maya, we have to ensure that we only use the following user input events :

left mouse button
middle mouse button
control modifier
shift modifier
backspace/delete to undo the last action
escape to exit the tool
enter to complete

The rest of the user input events are reserved for hotkeys, camera movement and marking menus. Any custom context we provide for our tools should therefore handle all user input events and rendering of any visual feedback needed.

 

 

   

 

 

To start with then, we will create our own rectangular selection tool. The class RectangularSelectTool derives from MPxContext and provides some custom user input processing.

 

 

 


 #ifdef WIN32
     #define NT_PLUGIN
 #endif  

 #include <maya/MStatus.h>
 #include <maya/MPxContext.h>
 #include <maya/MEvent.h>
 #include <maya/M3dView.h>

 class RectangleSelectTool
: public MPxContext
{
public:
/// \brief ctor
RectangleSelectTool();
/// \brief dtor
virtual ~RectangleSelectTool();
/// \brief This is used to setup any user interface things you need when the tool
/// is made current.
/// \param event - information on the input event
///
virtual void toolOnSetup( MEvent& event );
/// \brief overide to handle a mouse press event
/// \param event - information on the input event
/// \return MS::kSuccess or MS::kFailure
///
virtual MStatus doPress( MEvent& event );
/// \brief overide to handle a mouse drag event
/// \param event - information on the input event
/// \return MS::kSuccess or MS::kFailure
///
virtual MStatus doDrag( MEvent & event );
/// \brief overide to handle a mouse release event
/// \param event - information on the input event
/// \return MS::kSuccess or MS::kFailure
///
virtual MStatus doRelease( MEvent & event );

private:
/// The location of the cursor when the mouse was first clicked
short m_Start[2];
/// The location of the cursor when the mouse gets released
short m_End[2];
/// The View we are doing the selection in.
M3dView m_View; /// A flag to indicate that this tool is currently selecting things... bool m_IsSelecting;
};

 

 

 

 

First we shall provide a function that sets up the user interface when the tool gets initialised. In this case we will simply set the text contained within maya's help line.

 

 

 


 const char* g_HelpString = "Click and drag a rectangular region to select";

 void RectangleSelectTool::toolOnSetup( MEvent& event )
{ // make sure we set the initial value of the flag m_IsSelecting = false; // set the help text in the maya help boxs setHelpString( g_HelpString );
}

 

 

 

When the mouse click occurs, our doPress function needs to determine the event that occurred. Each user input function is provided with an MEvent class which provides info on which user input events were triggered (remember only those events listed above).

Contained within our class is an M3dView member variable, m_View. This is used to store reference to the 3D view in which the tool is actually being executed. The line

m_View = M3dView::active3dView();

Stores reference to this view so we can use it later.

The final thing that this function does is to get the location of the mouse cursor when it is clicked. By setting both the start and end points to the clicked point returned from event.getPosition, we now have a zero sized selection area.

 

 

 


 MStatus RectangleSelectTool::doPress( MEvent& event )
{
// if we have a left mouse click, start the selection of
// our rectangular region
//

if(event.mouseButton() == MEvent::kLeftMouse)
{
// We need to keep a reference of the view in which this command
// was called. This basically lets us know which viewport to draw
// in...

m_View = M3dView::active3dView(); // get the current position of the cursor. We need to know this so we can
// start drawing our selection rectangle!

event.getPosition( m_Start[0], m_Start[1] ); // make sure both the start and end points are in the same place.
m_End[0] = m_Start[0];
m_End[1] = m_Start[1]; // set flag to indicate that we are selecting things m_IsSelecting = true;
// yay!
return MS::kSuccess;
} // just let the base class handle the event
return MPxContext::doPress(event);
}

 

 

 

Having registered the start of the rectangular region using the doPress function, we can now handle the movement of the mouse by using the doDrag function.

To ensure that dragging the mouse with the middle mouse button will not affect our selection, we have to check our flag m_IsSelecting. If this flag is set, then we can update the size of our selection region.

Quite simply, the 2D coord m_Start gets set within the press function, as the mouse is being dragged we need to update the 2D coord m_End. By doing this we have a start and end point of a rectangle which can be displayed within our own overlay using simple openGL calls.

 

 

 


 MStatus RectangleSelectTool::doDrag( MEvent& event )
{
// if we have a left mouse click, start the selection of
// our rectangular region
//

if(m_IsSelecting)
{
// get the new location of the cursor

event.getPosition( m_End[0], m_End[1] ); // to draw the selection rectangle within maya, we need to
// draw a custom overlay in the viewport in which the tool
// is being used.
// // This means we have to begin and end the rendering of an overlay // between which we can include all that we want to draw //

m_View.beginOverlayDrawing(); // clear the overlay planes
m_View.clearOverlayPlane(); // Set up the orthographic projection matix
glMatrixMode( GL_PROJECTION );
glLoadIdentity();
gluOrtho2D( 0.0f, m_View.portWidth(),
0.0f, m_View.portHeight() );
glMatrixMode( GL_MODELVIEW );
glLoadIdentity(); // Set line width
glLineWidth( 1.0 ); // Draw rectangle
glBegin( GL_LINE_LOOP );
glVertex2i( m_Start[0], m_Start[1] );
glVertex2i( m_End[0], m_Start[1] );
glVertex2i( m_End[0], m_End[1] );
glVertex2i( m_Start[0], m_End[1] );
glEnd(); // done drawing our overlay on the viewport
m_View.endOverlayDrawing();
}
// just let the base class handle the event

return MPxContext::doDrag(event);
}

 

 

 

For our simple selection tool, most of the actual 'doing' part will occur when the mouse button is released. First we have to ensure we only react to the left mouse button, so we test that and return if the event is anything else.

During our doDrag implimentation, we rendered a rectangle over the viewport as an overlay. Having completed the tool, we really want to clear the overlay to return Maya to normal. To do this we simply need to call m_View.clearOverlayPlane().

Finally then, we need to actually make a selection. Normally within your own openGL programs this is a fairly complex affair. Happily within Maya, this is actually very simple indeed - we simply need to call MGlobal::selectFromScreen().

Now, one small caveat! We have simply been storing the start and end points of the mouse clicks. The selectFromScreen() method requires the minimum and maximum points that define the selection area. Since we can select in any direction across our context, we will need to determine the minimum and maximum extents of our selection tool.

 

 

 


 MStatus RectangleSelectTool::doRelease( MEvent& event )
{ // only bother handling the release of a left mouse button.
if(event.mouseButton() != MEvent::kLeftMouse) {
return MPxContext::doRelease(event);
} // reset our flag cos we are no longer selecting m_IsSelecting = false; // Clear the overlay plane & restore from overlay drawing
m_View.beginOverlayDrawing();
m_View.clearOverlayPlane();
m_View.endOverlayDrawing(); // since the selection area can be dragged in -ve x and -y axes,
// we need to allow for this and calculate a min and max point
// (for that is what the selection mechanism will need)
//
short MinX,MinY;
short MaxX,MaxY; // if the selection went left if(m_End[0] < m_Start[0]) {
MinX = m_End[0];
MaxX = m_Start[0];
}
else {
MaxX = m_End[0];
MinX = m_Start[0];
}
// if the end y is less than the start y if(m_End[1] < m_Start[1]) {
MinY = m_End[1];
MaxY = m_Start[1];
}
else {
MaxY = m_End[1];
MinY = m_Start[1];
} // Use the selection mechanism within MGlobal to select the items within the viewport
//
MGlobal::selectFromScreen( MinX, MinY, MaxX, MaxY, MGlobal::kReplaceList ); // nullify selection area
m_Start[0] = m_Start[1] = m_End[0] = m_End[1] = 0; return MS::kSuccess;
}

 

 

 

 

In addition to providing a context for our tool, we have to additionally provide a context command.

Consider, a context is something that is associated with ONE of the 3D views in Maya. However, when we enter a specific tool, we can use it in ANY of the 3D views.

When we enter our tool, an instance of our context command is created. When we click within one of the views Maya calls the virtual function makeObj(). This returns a new instance of the context to use.

If this is in any way confusing, don't worry. We don't register the context, we register a command to create a context. It's always the same as this... so just know you need to do it this way ;)

 

 

 


 class RectangleSelectToolCommand
: public MPxContextCommand
{
public:
/// \brief This function needs to return a context for maya to use
/// when our tool requires input from the viewport.
/// \return a new instance of RectangleSelectTool
///

virtual MPxContext* makeObj(); /// \brief This returns a new instance of the tool command for maya to use.
/// \return an instance of RectangleSelectContextCmd
///

static void* creator();
}; MPxContext* RectangleSelectToolCommand::makeObj()
{
return new RectangleSelectTool;
} void* RectangleSelectToolCommand::creator()
{
return new RectangleSelectToolCommand;
}

 

 

 

So then, to registering our context... as with all plug-ins, we use the Plugin function set to register our context command...

 

 

 


 #include "RectangleSelectTool.h"
 #include <maya/MFnPlugin.h>


 // This is a nasty bit of hackyness for compilation under Windows. Under Win32 you need 
 // to compile a dll project and change the extension from "dll" to "mll". One additional
 // thing we have to do is 'export' the initializePlugin and uninitializePlugin functions. 
 // This basically means that when maya loads the dll, it can see the two Functions it needs.
 // If for some reason your plugin fails to load, it may be this thats causing the problems.
 // Under linux we simply need to compile it with the -shared flag.
 // 
 #ifdef WIN32
     #define MLL_EXPORT __declspec(dllexport) 
 #else
     #define MLL_EXPORT
 #endif  
 
 //------------------------------------------------------------------- 
 /// \brief initializePlugin( MObject obj )
 /// \param obj - the plugin handle
 /// \return MS::kSuccess if ok 
 /// \note Registers all of the new commands, file translators and new
 /// node types. 
 /// 
 MLL_EXPORT MStatus initializePlugin(MObject obj ) { 
 	MFnPlugin plugin( obj, "Rob Bateman", "1.0", "Any");  

	// register the mel command with the plugin function set.
	// Do this for each mel command your plugin is going to add into Maya
	//
status = fnPlugin.registerContextCommand( "RectangleSelectToolContext",
RectangleSelectToolCommand::creator ); if (!status) { status.perror("Failed to register \"RectangleSelectToolContext\"\n"); return status; } return status; } //------------------------------------------------------------------- /// \brief uninitializePlugin( MObject obj ) /// \param obj - the plugin handle to un-register /// \return MS::kSuccess if ok /// \note un-registers the plugin and destroys itself /// MLL_EXPORT MStatus uninitializePlugin( MObject obj ) { MFnPlugin plugin( obj ); // deregister the mel command with the plugin function set // Do this for each mel command your plugin has added into Maya. // status = plugin.deregisterContextCommand( "RectangleSelectToolContext" ); if (!status) { status.perror("failed to deregister \"RectangleSelectToolContext\"\n"); return status; } return status; }

 

 

 

After the plugin is loaded, we need to be able to tell maya to use our tool. To do this, run the following mel commands...

 

 

 

 // creates an instance of our tool
 RectangleSelectToolContext;

 // this will produce the tool instance, RectangleSelectToolContext1.
 // simply tell maya to use this tool...
 setToolTo RectangleSelectToolContext1;

 

 

 

Before unloading the plugin, you should first destroy the tool by typing the following commands (later we shall automate this).

 

 

 

 // destroy the instance of our tool
 deleteUI -tc RectangleSelectToolContext1;

 

 

 

 

Well, if you've played about with the plugin a bit, you may have noticed one or two flaws compared to Maya's default selection tool.

1. Selection only works in the 3D views, ie, you cannot select UV coords within the UV texture editor.

2. The modifiers control and shift have no affect. Normally ctrl+select removes items from the selection list; shift+select toggles items in the selection list and ctrl+shift+select adds items to the selection list.

3. No icon for our tool appears within the Toolbox on the left of maya.

4. Typing setToolTo is a bit long winded. We could do with an icon or some user interface item that sets the tool for us.

 

Unfortunately, I have found no way to be able to select items within anything other than the 3D views. If anyone finds a way, I'd be very interested in hearing it....

So then, we may as well start on fixing the modifier problem...

 

 

 

Task1: Using The Modifiers Control & Shift

 

 

 

 

Within MGlobal, the enum MGlobal::ListAdjustment defines the following values to determine the selection operation that will occur.

MGlobal::kReplaceList
MGlobal::kXORWithList
MGlobal::kAddToList
MGlobal::kRemoveFromList

Therefore, we can store a ListAdjustment variable within our class to see what operation should be performed...

 

 

 

 


 #ifdef WIN32
     #define NT_PLUGIN
 #endif  

 #include <maya/MStatus.h>
 #include <maya/MPxContext.h>
 #include <maya/MEvent.h>
 #include <maya/M3dView.h>

 class RectangleSelectTool
: public MPxContext
{
public:
/// \brief ctor
RectangleSelectTool();
/// \brief dtor
virtual ~RectangleSelectTool();
/// \brief This is used to setup any user interface things you need when the tool
/// is made current.
/// \param event - information on the input event
///
virtual void toolOnSetup( MEvent& event );
/// \brief overide to handle a mouse press event
/// \param event - information on the input event
/// \return MS::kSuccess or MS::kFailure
///
virtual MStatus doPress( MEvent& event );
/// \brief overide to handle a mouse drag event
/// \param event - information on the input event
/// \return MS::kSuccess or MS::kFailure
///
virtual MStatus doDrag( MEvent & event );
/// \brief overide to handle a mouse release event
/// \param event - information on the input event
/// \return MS::kSuccess or MS::kFailure
///
virtual MStatus doRelease( MEvent & event );

private:
/// The location of the cursor when the mouse was first clicked
short m_Start[2];
/// The location of the cursor when the mouse gets released
short m_End[2];
/// The View we are doing the selection in.
M3dView m_View; /// A flag to indicate that this tool is currently selecting things... bool m_IsSelecting; /// The selection mode determine by the modifiers shift and control *NEW* MGlobal::ListAdjustment m_Mode;
};

 

 

 

 

We will now have to update our doPress function so that we check to see what modifiers were pressed and store the mode for use later on.

 

 

 


 MStatus RectangleSelectTool::doPress( MEvent& event )
{
// if we have a left mouse click, start the selection of
// our rectangular region
//

if(event.mouseButton() == MEvent::kLeftMouse)
{ //------------------------------------------------------- *NEW* // if both held, then add to list if( event.isModifierShift() && event.isModifierControl() ) { m_Mode = MGlobal::kAddToList; } else // if toggling selection with shift if( event.isModifierShift() ) { m_Mode = MGlobal::kXORWithList; } else // if removing from the selection list if( event.isModifierControl() ) { m_Mode = MGlobal::kRemoveFromList; } // default: replace selection list else { m_Mode = MGlobal::kReplaceList; } //------------------------------------------------------- *NEW*
// We need to keep a reference of the view in which this command
// was called. This basically lets us know which viewport to draw
// in...

m_View = M3dView::active3dView(); // get the current position of the cursor. We need to know this so we can
// start drawing our selection rectangle!

event.getPosition( m_Start[0], m_Start[1] ); // make sure both the start and end points are in the same place.
m_End[0] = m_Start[0];
m_End[1] = m_Start[1]; // set flag to indicate that we are selecting things m_IsSelecting = true;
// yay!
return MS::kSuccess;
} // just let the base class handle the event
return MPxContext::doPress(event);
}

 

 

 

Within the doRelease function, we now have to make use of the toggled selection mode. This is simply a case of ammending the call to select from the screen, so that it uses our stored mode, ie :

MGlobal::selectFromScreen( MinX, MinY, MaxX, MaxY, m_Mode);

If you compile the plugin and try it out, we should now have modifiers affecting what we select.

 

 

 

 

 

Task2: Adding A User Interface

 

 

 

Ok, on now to the user interface. First off, we will create a mel script called "RectangularSelectToolCreateUI.mel" and save it within our ~maya/scripts directory. Later on we will set this to be called automatically by our plugin...

You will notice the use of the mel command optionVar within the script. This command lets us store our own options within maya. We can use this when we want to save any user settings even if maya shuts down - generally nicer for this sort of thing that global script variables!!

The command can be used with the -exists flag to determine if the specified option exists; the -sv to save the specified data, or the -rm flag to remove the option.

The bulk of this mel script is used to create our context, and then create a tool button for it on the Custom shelf. You will also notice that we also specify a name for an icon to use. To create your own icons, you can create any image you want in gimp and save it as a 32x32 RGB xpm file.

 

 

 


 //
 // RectangleSelectToolCreateUI.mel
 //

 global proc RectangleSelectToolCreateUI()
{
// check to see if an option already exists

if (`optionVar -exists SelectToolContextName`) {
return;
} // create the tool
string $SelectToolContextName = `eval "RectangleSelectToolContext"`; // create a tool button on the "Custom" shelf
string $SelectToolButtonName = `toolButton
-doubleClickCommand "toolPropertyWindow"
-cl toolCluster
-p Custom // the name of the shelf for the button
-t $SelectToolContextName // the name of the context command
-i1 "RectangleSelectTool.xpm"`; // the icon name // Save the names of the UI objects just created.
optionVar -sv SelectToolContextName $SelectToolContextName
-sv SelectToolButtonName $SelectToolButtonName;
}

 

 

 

The UI deletion script simply deletes the button and the context. The user setting variables are also deleted (optionVar -rm).

 

 

 


 //
 // RectangleSelectToolDeleteUI.mel
 //

 global proc RectangleSelectToolDeleteUI()
{
// just so we remain on the cautious side, set the tool to the main select tool.
// this just ensures we don't unload the tool whilst it is being used...
//

setToolTo selectSuperContext; // if the button exists as a maya option
if( `optionVar -exists SelectToolButtonName` )
{
// query the select tool button name

string $SelectToolButtonName = `optionVar -q SelectToolButtonName`;

// delete it

deleteUI $SelectToolButtonName;

// remove the option string

optionVar -rm SelectToolButtonName;
} // do the same for the tool's context
if( `optionVar -exists SelectToolContextName` )
{
string $SelectToolContextName = `optionVar -q SelectToolContextName`;
deleteUI -tc $SelectToolContextName;
optionVar -rm SelectToolContextName;
}
}

 

 

 

As a small addition, we will also make a couple of changes to the toolOnSetup function to specify the icon for the toolbox, and the title within the tool options.

 

 

 


 const char* g_HelpString = "Click and drag a rectangular region to select";

 void RectangleSelectTool::toolOnSetup( MEvent& event )
{ // make sure we set the initial value of the flag m_IsSelecting = false; // set the help text in the maya help boxs setHelpString( g_HelpString ); // set the Title in the Tool Options
setTitleString ( "Rectangle Selection Tool" ); // set the icon that will appear in the toolbox
setImage("RectangleSelectTool.xpm", MPxContext::kImage1 );
}

 

 

 

 

Right then, as a final step, we can actually use the MFnPlugin function set to register UI creation and deletion scripts. The only change we have to now make is when we initialise the plugin.

To do this we can make use of the MFnPlugin::registerUI function. This simply takes the name of 2 scripts, one to be called when the plugin is loaded, the second to be called when it's unloaded.

 

 

 


 MLL_EXPORT MStatus initializePlugin(MObject obj ) { 
 	MFnPlugin plugin( obj, "Rob Bateman", "1.0", "Any");  

	// register the mel command with the plugin function set.
	// Do this for each mel command your plugin is going to add into Maya
	//
status = fnPlugin.registerContextCommand( "RectangleSelectToolContext",
RectangleSelectToolCommand::creator ); if (!status) { status.perror("Failed to register \"RectangleSelectToolContext\"\n"); return status; } // set the mel scripts to be run when the plugin is loaded / unloaded
status = fnPlugin.registerUI("CircleSelectToolCreateUI", "CircleSelectToolDeleteUI"); // check error
if (!status) {
status.
perror("registerUIScripts");
return status;
}
return status; }