MKV file in Mac

.MKV is a container format developed by Matroska .  It can hold different types of video and audio. For example, there might be an .AVI file contained within the .MKV file.
The Matroska Multimedia Container is an open standard free Container format, a file format that can hold an unlimited number of video, audio, picture or subtitle tracks inside a single file. It is intended to serve as a universal format for storing common multimedia content, like movies or TV shows. Matroska is similar in conception to other containers like AVI, MP4 or ASF, but is entirely open in specification, with implementations consisting mostly of open source software. Matroska file types are .MKV for video (with subtitles and audio), .MKA for audio-only files and .MKS for subtitles only. High Definition movies in 720 or 1080 pixel width format are often encoded and packaged as a Matroska format video with a .mkv file extension.

MPlayer OS X and VLC Media Player both are free cross-platform multimedia player.

These excellent and free multimedia players can tackle many of the video formats unknown to QuickTime. They contain the decoder to decode the .mkv files. They will also play many of the formats you will find inside the .mkv file.

The totally free video converter, iSquint  developed by Techspanion, can convert MKV to iPod (Nano, Classic, touch), convert MKV to iPhone, Apple TV, Xbox 360, PSP, PS3, AVI, MP4, MPEG, Flash and WMV. But Techspanion has been discontinue, you can download the final version on macupdate.com.
And after an outcry from fans, iSquint, the popular video converter on Mac OS X have now gone open source. It has got new name Film Redux. You can download its open source and built it by yourself.