Fun with code (since 2006)

Digging more into the Molehill APIs

01.06.11 Posted in 3D, Actionscript 3, Inside the Flash Player, Molehill by

A few months ago, we announced at Max 2010 in Los Angeles, the introduction of the Molehill APIs in the Adobe Flash runtimes on mobile and desktop. For more infos check the “Molehill” page on Adobe Labs. I wanted to give you guys more details about Molehill, some more technical details on how it is going to work from an ActionScript developer standpoint.

So let's get started ;)

What is Molehill?

“Molehill’ is the codename for the set of 3D GPU accelerated APIs that will be exposed in ActionScript 3 in the Adobe Flash Player and Adobe AIR. This will enable high-end 3D rendering inside the Adobe Flash Platform. Molehill will rely on DirectX9 on Windows, OpenGL 1.3 on MacOS and Linux. On mobile platforms like Android, Molehill will rely on OpenGL ES2. Technically, the Molehill APIs are truly 3D GPU programmable shader based, and will expose features that 3D developers have been looking for since a long time in Flash, like programmable vertex and fragment shaders, to enable things like vertex skinning on the GPU for bones animation but also native z-buffering, stencil color buffer, cube textures and more.

In terms of performance, Adobe Flash Player 10.1 today, renders thousands of non z-buffered triangles at approximately 30 Hz. With the new 3D APIs, developers can expect hundreds of thousands of z-buffered triangles to be rendered at HD resolution in full screen at around 60 Hz. Molehill will make it possible to deliver sophisticated 3D experiences across almost every computer and device connected to the Internet. To get an idea of how Molehill performs and see a live demo check this video.

The way it works.

The existing Flash Player 2.5D APIs that we introduced in Flash Player 10 are not deprecated, the Molehill APIs will offer a solution to advanced 3D rendering requiring full GPU acceleration. Depending on the project that you will be working on, you will decide which APIs you want to use.

We introduced recently the concept of “Stage Video” in Flash Player 10.2 available as a beta on Adobe Labs.
Stage Video relies on the same design, by enabling full GPU acceleration for video, from decoding to presentation. With this new rendering model, the Adobe Flash Player does not present the video frames or 3D buffer inside the display list but inside a texture sitting behind the stage painted through the GPU. This allows the Adobe Flash Player to directly paint on screen the content available on the graphics card memory. No more read back is required, to retrieve the frames from the GPU to push them on screen through the display list on the CPU.

As a result, because the 3D content sits behind the Flash Player stage and is not part of the display list, the Context3D and Stage3D objects are not display objects. So remember that you cannot interact with them just like with any DisplayObject, rotations, blend modes, filters and many other effects cannot be applied.

The following figure illustrates the idea:

Stage3D Model

Of course, as you can see, 2D content can overlay the 3D content with no problems, but the opposite is not possible. However, we will provide an API which will allow you to draw your 3D content to a BitmapData if required. From an ActionScript APIs standpoint, as a developer you interact with the two main objects, a Stage3D and a Context3D object. You request to the Adobe Flash Player a 3D context and a Context3D object will be created for you. So now you may wonder, what happens if the GPU driver is incompatible, do I get a black screen failing silently?

The Flash Player will still return you a Context3D object but using software fallback internally, so you will still get all the Molehill features and same API but running on the CPU. To achieve, this we rely on a very fast CPU rasterizer from TransGaming Inc. called “SwiftShader”. The great news is that even when running on software, SwiftShader runs about 10 times faster than today’s vector rasterizer available in Flash Player 10.1, so you can expect some serious performance improvements even when running in software mode.

The beauty of “Molehill” APIs is that you do not have to worry what is happening internally. Am I running on DirectX, OpenGL or SwiftShader? Should I use a different API for OpenGL when on MacOS and Linux or OpenGL ES 2 when running on a mobile platform? No, everything is transparent for you as a developer, you program one single API and the Adobe Flash Player will handle this for you internally and do the translation behind the scene.

It is important to remember, that the Molehill APIs do not use what is called a fixed function pipeline but a programmable pipeline only, which means that you will have to work with vertex and fragment shaders to display anything on screen. For this, you will be able to upload on the graphics card your shaders as pure low-level AGAL (“Adobe Graphics Assembly Language”) bytecode as a ByteArray. As a developer you have two ways to do this, write your shaders at the assembly level, which requires an advanced understanding of how shaders works or use a higher-level language like Pixel Bender 3D which will expose a more natural way to program your shaders and compile for you the appropriate AGAL bytecode.

In order to represent your triangles, you will need to work with VertexBuffer3D and IndexBuffer3D objects by passing vertices coordinates and indices, and once your vertex shaders and fragment shaders are ready, you can upload them to the graphics card through a Program3D object. Basically, a vertex shader deals with the position of the vertices used to draw your triangles whereas a fragment shader handles the appearance of the pixels used to texture your triangles.

The following figure illustrates the difference between the types of shaders:



As stated before, Molehill does not use a fixed function pipeline, hence developers will be free to create their own custom shaders and totally control the rendering pipeline. So let’s focus a little bit on the concept of vertex and fragment shaders with Molehill.

Digging into vertex and fragment shaders

To illustrate the idea, here is a simple example of low-level shading assembly you could write to display your triangles and work at the pixel level with Molehill. Let's get ready, cause we are going to go very low-level and code shaders to the metal ;). Of course if you hate this, do not worry, you will be able to use a higher-level shading language like Pixel Bender 3D.

Note : To compile the assembly String to AGAL bytecode, download the AGALMiniAssembler here.

To create our shader program to upload to the graphics card; we need first a vertex shader (Context3DProgramType.VERTEX) which should at least output a clip-space position coordinate. To perform this, we need to multiply va0 (each vertex position attributes) by vc0 (vertex constant 0) our projection matrix stored at this index and output the result through the op keyword (standing for "output position" of the vertex shader):

// create a vertex program - from assembly
var vertexShaderAssembler : AGALMiniAssembler = new AGALMiniAssembler();

vertexShaderAssembler.assemble( Context3DProgramType.VERTEX,
"m44 op, va0, vc0 \n" // 4x4 matrix transform from stream 0 (vertex position) to output clipspace

Now you may wonder, what is this m44 thing? Where does it comes from?

It is actually a 4x4 matrix transformation, it projects our vertices according to the projection matrix we defined, we could have written our shader like the following, by manually calculating the dot product on each attribute, but the m44 instruction (performing a 4x4 matrix transform on all attributes in one line) is way shorter:

// create a vertex program - from assembly
var vertexShaderAssembler : AGALMiniAssembler = new AGALMiniAssembler();

vertexShaderAssembler.assemble( Context3DProgramType.VERTEX,
"dp4 op.x, va0, vc0 \n" + // 4x4 matrix transform from stream 0 (vertex position) to output clipspace
"dp4 op.y, va0, vc1 \n" +
"dp4 op.z, va0, vc2 \n" +
"dp4 op.w, va0, vc3 \n" +

Remember that vc0 (vertex constant 0), it is actually just our projection matrix stored in this index, passed earlier as a constant through the setProgramsConstantMatrix API on the Context3D object:

context3D.setProgramConstantsFromMatrix(Context3DProgramType.VERTEX, 0, modelMatrix, true );

As with our matrix constant, va0 (vertex attributes 0) for the position, needs to be defined, and we did this through the setVertexBufferAt API on the Context3D object.

context3D.setVertexBufferAt (0, vertexbuffer, 0, Context3DVertexBufferFormat.FLOAT_3 );

In our example, the vertex shader passes the vertices color (va1) to the fragment shader through v0 and the mov instruction to actually paint our triangles pixels. To do this, we could write the following:

// create a vertex program - from assembly
var vertexShaderAssembler : AGALMiniAssembler = new AGALMiniAssembler();

vertexShaderAssembler.assemble( Context3DProgramType.VERTEX,
"m44 op, va0, vc0 \n" +	// 4x4 matrix transform from stream 0 (vertex position) to output clipspace
"mov v0, va1 \n"  	// copy stream 1 (vertex color) to fragment shader

And as you can imagine, va1 (vertex attributes 1) for the color was defined through setVertexBufferAt, to expose our pixel colors (float 3) in the shaders:

context3D.setVertexBufferAt( 1, vertexbuffer, 3, Context3DVertexBufferFormat.FLOAT_3 );

Our vertices position and colors are defined into our VertexBuffer3D object :

// create a vertex buffer
// format is (x,y,z,r,g,b) = 3 vertices, 6 dwords per vertex
vertexbuffer.uploadFromVector ( Vector.<Number>([
-1,-1,0,  255/255,0,0, 				// red
0,1,0,    193/255,216/255,47/255, 	// green
1,-1,0,   0,164/255,228/255       	// blue
]),0, 3 ); // start at offset 0, count 3

We have our vertex shader defined, now we need to define and upload our fragment shader (Context3DProgramType.FRAGMENT), the idea is to retrieve each vertex color passed (copied from va1 to v0) and output this color through the oc opcode:

var fragmentShaderAssembler : AGALMiniAssembler= new AGALMiniAssembler();
fragmentShaderAssembler.assemble( Context3DProgramType.FRAGMENT,
"mov oc, v0" // output color

As you can imagine, a fragment shader should always output a color. Then, we need to upload all this to the Context3D object:

// upload the AGAL bytecode
program = context3D.createProgram();
program.upload( vertexShaderAssembler.agalcode, fragmentShaderAssembler.agalcode );

If we compile and run those shaders, we would get the following result:

Hello Triangle

Now, let’s say we need to invert the colors of each pixel, it would be really easy. As this operation is performed on the pixels color only, we would just modify our fragment shader and use the sub opcode to subtract the color, as following:

var fragmentShaderAssembler : AGALMiniAssembler= new AGALMiniAssembler();
fragmentShaderAssembler.assemble( Context3DProgramType.FRAGMENT,
"sub ft0, fc1, v0 \n" + // subtract the color ( 1 - color)
"mov oc, ft0" // output color

Here, we invert the color of each pixel by subtracting each pixel color from 1 (white). The white pixel we subtract from is stored in a fragment constant (fc1) that we passed by using the setProgramConstantsFromVector API:

context3D.setProgramConstantsFromVector(Context3DProgramType.FRAGMENT, 1, Vector.( [ 1, 1, 1, 1 ] ) );

The final pixel color is then stored in a fragment temporary register (ft0) and passed as the final output color.

By using this modified fragment shader, we end up with the following result:

Hello Triangle Inverted

As another exercice, let's process a sepia-tone filter.

To achieve this, we need to first convert to gray scale then to sepia. We would use the following fragment shader for this:

var fragmentShaderAssembler : AGALMiniAssembler= new AGALMiniAssembler();
fragmentShaderAssembler.assemble( Context3DProgramType.FRAGMENT,
"dp3 ft0, fc1, v0  \n" + // convert to grayscale
"mul ft1, fc2, ft0 \n" + // convert to sepia
"mov oc, ft1" // output color

As usual, we would have defined our constants using the setPrograConstantsFromVector API:

// grayscale
context3D.setProgramConstantsFromVector(Context3DProgramType.FRAGMENT, 1, Vector.<Number>( [ 0.3, 0.59, 0.11, 1 ] ) );
// sepia
context3D.setProgramConstantsFromVector(Context3DProgramType.FRAGMENT, 2, Vector.<Number>( [ 1.2, 1.0, 0.8, 1 ] ) );

By using such a fragment shader, we would end up with the following result:

Hello Triangle Sepia

As you can imagine, this gives you a lot of power and will allow you to go way further in terms of shading and handle things like lightning through vertex or fragment shading, fog, or even animation through vertex skinning and even more.

Ok, so last one, let's now apply a texture to our triangle from a BitmapData, to do this, we would need to pass uv values from our vertex shader to our fragment shader and then use those values to apply our texture that we sampled in our fragment shader.

To pass the uv values, we would need to modify our vertex shader this way :

vertexShaderAssembler = new AGALMiniAssembler();
vertexShaderAssembler.assemble( Context3DProgramType.VERTEX,
"m44 op, va0, vc0    \n" + // 4x4 matrix transform from stream 0 to output clipspace
"mov v0, va1         \n"   // copy texcoord from stream 1 to fragment program

Our uv coordinates are now copied from va1 to v0, ready to be passed to the fragment shader. Notice that we do not pass any vertex color anymore to the fragment shader, just the uv coordinates.

As expected, we defined our uv values for each vertex (float2) through va1 with setVertexBufferAt :

context3D.setVertexBufferAt( 1, _vertexBuffer, 2, Context3DVertexBufferFormat.FLOAT_2 );

Our vertices position and uv values are defined into our VertexBuffer3D object :

// x,y,u,v
-1,-1, 0,1,
0,1, 1,0,
1,-1, 1,1,
0, 3

Then we retrieve the uv values in our fragment shader and sample our texture :

fragmentShaderAssembler.assemble( Context3DProgramType.FRAGMENT,
"mov ft0, v0 \n"+
"tex ft1, ft0, fs1 <2d,clamp,linear> \n"+ // sample texture 1
"mov oc, ft1 \n"

To define our texture, we instantiate our BitmapData, upload it to a Texture object and upload it to the GPU:

texture = context3D.createTexture( 256, 256, Context3DTextureFormat.BGRA, false );
var bitmap:Bitmap = new MolePeopleBitmap();
texture.uploadFromBitmapData( bitmap.bitmapData );

And then to access it from fs1, we set it:

context3D.setTextureAt( 1, texture );

By using this modified shader program, we end up with this :

Textured Triangle

I will cover in later tutorials new effects like per-fragment fog or heat signature with texture lookup too.

Of course, we just covered here how shaders work with Molehill. To control your pixels you need triangles and vertices and indices defining them in your scene. For this, you will need other objects like VertexBuffer3D and IndexBuffer3D, attached to your Context3D object.

The following figure illustrates the overall interaction of objects:

Molehill Architecture

As you can see, the Molehill APIs are very low-level and expose features for advanced 3D developers who want to work at such a level with 3D. Of course, some developers will prefer working with higher-level frameworks, which exposes ready to go APIs, and we took care of that.

Building mountains out of Molehill

We know that many ActionScript 3 developers would prefer having a light, a camera, a plane to work with rather than a vertex buffer and shaders bytecode. So to make sure that everyone can enjoy the power of Molehill, we are actively working with existing 3D frameworks like Alternativa3D, Flare3D, Away3D, Minko, Sophie3D, Yogurt3D and more. Today most of these frameworks are already Molehill enabled and will be available for you when Molehill is available in a next version of the Adobe Flash runtimes.

Most of the developers from these frameworks were at Max this year to present sessions about how they leveraged Molehill in their respective framework. We expect developers to build their engine on top of Molehill, hence, advanced 3D developers and non-3D developers will be able to benefit from Molehill.

I hope you enjoyed this little deep dive into Molehill, stay tuned for more Molehill stuff soon ;)

93 Responses to “Digging more into the Molehill APIs”

  1. [...] 深入挖掘Molehill API的一些特征(翻译版):,原文: [...]

  2. [...] as possible in the code so you can follow along. I should note that this source code is based on Thibault Imbert’s blog post and this video. [...]

  3. [...] talking about and you’re have heard the word pixelshader for the first time, take a look at Thibault’s intro to Molehill and Michael’s Simple 2D Molehill [...]

  4. [...] 元記事はコチラ。 [...]

  5. alijaya says:

    hmmm… i think you have forgotten to write
    context3D.setVertexBufferAt( 0, _vertexBuffer, 0, Context3DVertexBufferFormat.FLOAT_2 );
    when you tell how to use bitmapdata
    I think it must FLOAT_2, but the article doesn’t note it, it seems it must be FLOAT_3

    am i right? :-?

  6. [...] If you want to better understand exactly what Molehill is, I suggest reading Thibault Imbert’s recent blog post, Digging more into the Molehill APIs [...]

  7. [...] 深入挖掘Molehill API的一些特征(翻译版):,原文: [...]

  8. Zhen Ju says:

    I’m having some troubles running MaxRacer, my graphics card is Radeon 5770, supports dx9, dx10 and even dx11, but when I’m runing maxracer, the CPU accupation is very high(over 90%), it’s obvious that the GPU is not working with maxracer, how can i solve the problem? (waiting for you advice ^_<)

  9. Zhen Ju says:

    My video card is Radeon 5770, it can support DX9, but when i’m running maxracer, the cpu is very busy(over 90% by the game), it’s obvious that the video card is not working, is there some way i can solve this problem?

  10. [...] If you want to better understand exactly what Molehill is, I suggest reading Thibault Imbert’s recent blog post, Digging more into the Molehill APIs [...]

  11. [...] to the triangles which are drawn to the screen.  As the Molehill APIs are very low level, your only tool for writing shaders is Assembley (via AGAL (Adobe Graphics Assembly Language)) which is something most FlashDevs won’t be [...]

  12. Zhen Ju says:

    Hey Thibault,
    I’m trying to test your code in this paper, I’ve installed the fp11 reqired tools(fp11.xml,playerglobal.swfc), all thoese needed for flash ide professional. But I encounter a strange problem! It reminds me that flash.display3D::Context3D can not be found! But I’ve put the files into the right place(I’m having the correct code hints and fp11 publish settings), is there something more I should do?

  13. Zhen Ju says:

    Hey Thibault,
    I’ve solved the previous problem! It’s because we have’nt yet had a standalone flash player 11, so i should try this program in web browsers! I’m trying to write the code in my flash cs5, and then use publish preview(f12) to test it~ well, it’s a tough job for me to checkout the documentation[I'm Chinese, not very good at English :(], but i’m gonna try hard!

  14. Pleh says:

    Hi Thibault,

    Not sure if you know the answer to this or not but I thought I would ask anyway…

    Can you use molehill with the flash packager for iphone? Or are there any plans to enable this?


  15. Michiel Brinkers says:

    Will the AGAL compiler eventually be integrated into the flash player/playerglobals? Seems like functionality which would benefit from native implementation.

  16. BillO says:

    I heard that Unity was developing an export for Flash that utilizes this new technology. Unity has a browser plug-in, but being able to use Unity and the Flash player is a much better solution.

  17. Michiel Brinkers says:

    Thibault, can you maybe explain why Context3D.drawToBitmapData is so slow? 33 ms for copying a 256*256 image buffer seems rather long.
    Is there some problem with copying from the video memory back to RAM? Because once it’s in RAM I would expect the copy to a BitmapData to be really fast.

  18. [...] signal processing. Using Pixel Bender 3D or AGAL (“Adobe Graphics Assembly Language”) you can write your own shaders to crunch bytes fast. Whether those bytes represent 3D vertices, 2D sprites or audio samples, Flash Player 11 and your [...]

  19. C’est tres prometteur tout ca! Je m’attaque tout de suite a MoleHill!

    Sourigna (EPITA 2011)

  20. [...] running was among the most eminent problems I had… the featured article “Digging more into the Molehill APIs by Thibault Imbert” on the molehill page is pretty useless. It describes what the next version will be able to [...]

  21. [...] 深入挖掘Molehill API的一些特征(翻译版):,原文: [...]

  22. [...] Understanding Molehill – Digging Deeper Into Molehill Understanding Molehill – Alternativa 3D 8 Migration (Talks about a few underlying concepts of [...]

  23. [...] Stage3D Stage3D là phần đứng giữa Flash và bộ máy xử lý đồ họa (GPU). Stage3D không nằm trong display list (bạn không thể làm thao tác addChild với Stage3D). Stage3D tồn tại song song với đối tượng Stage chúng ta đã biết. Stage3D nằm đằng sau tất cả các đối tượng hiển thị của Flash nhưng nằm đằng trước StageVideo. Minh họa Stage3D (nguồn: ByteArray) [...]

  24. [...] auch immer man dies tun sollte ). Außerdem greift Molehill im Falle einer fehlenden GPU auf einen Software Renderer namens SwiftShader zu, so dass immer eine Stage3D-Instanz vorhanden sein müsste. Ist eine GPU vorhanden, werden die [...]

  25. [...] of all there is plenty resources available over the web you can learn a basics. would be good starting point. There is also well written article by [...]

  26. Neil says:

    Without any framework wrapping this stuff it will be out of reach for most Flash Developers.

  27. Gildas says:

    Have you considered accessibility at all? Is there any concept of providing text equivalent assignments of object models that can be passed to the flash player and the operating system’s accessibility APIs?

  28. Kevin Burke says:

    Salut Thibault,
    I have a question – how do StageVideo & Stage3D work together? I’m looking to play a video with Stage Video and overlay 3D on top of it with an engine such as Away3D. If this is not possible, and Away3D needs to be software rendered, how can I use Flash 3D engines like this after they have been rewritten to use Stage3D? Merci!

  29. Thibault Imbert says:

    Hi Kevin,

    This is possible, you will notice that in FP11 public beta, we also introduced a new feature. Stage3D can now be transparent, meaning that you can have a StageVideo surface in the back and Stage3D on top completely HW accelerated.


  30. Kevin Burke says:

    Thank you Thibault, and that is AWESOME!!!
    I tried getting it to work, but I don’t think I’m doing it correctly (because Away3D’s bg is still white).

    for (var i:uint=0; i<stage.stage3Ds.length; i++)
    this.stage.stage3Ds[i].transparent = true;

  31. fourfire says:

    There is a very painful problem, when i lock my computer,then open it, my flashplayer11 throw a error :
    Error: Error #3694: The object was disposed by an earlier call of dispose() on it.
    track it by debug and find the Context3D is null, so i create other one by requestContext3D()method of Stage3D class,but there is a error, Error #3600 : No valid program set

  32. [...] of you, who had a look at the Stage3D APIs understand that this can be complex sometimes. So instead of writing 60 lines of code to draw a [...]

  33. [...] vs. using shaders and bytecode assemblers (which is awesome or horrible depending): [...]

  34. [...] shipping product. Sometimes, you have no choice but to tack it on top of (or in Flash’s case, behind) the existing [...]

  35. [...] 보기 전에 아래 글을 선행적으로 본다면 이해하는데 도움이 될 것이다. Digging more into the Molehill APIs MoleHill Getting [...]

  36. [...] been a long time I did anything with actual 3D modeling, the logical place to start is 2D. Dreading the complexity of the Stage3D API’s I will focus on the Starling framework. There is a nice example of what complexity Starling hides [...]

  37. [...] behind all flash content. If you want to get a little low level knowledge, read Thibault’s article here. Using the GPU, the flash player is able to render full screen HD content at 60hz… Finally a [...]

  38. [...] Digging more into the Molehill APIs – [...]

  39. [...] 源地址:Digging more into the Molehill APIs [...]


Leave a Reply

Open Sort Options

Sort comments by:
  • * Applied after refresh
Fun with code since 2006.