The icon shows in a file open/save dialog, but not in explorer. How can I get the icon to show in explorer toolbar as well?

I don't see anything...what is it?

You can't.

However, bxNewFolder can

I edited the post with a better image url.
Do I have to run a 3rd party program just to put the icon there? I would like it right after the 'up' icon as it is in the open/save dialog

Yes, you need to run a 3rd party app. You can place it next to the 'up' icon, but the next time you start the program it'll go back to its default position, which is near the corner.