We are never told what a gump was like. Its head resembled that of an elk with large antlers and a long beard on its chin. What its body was like is never revealed in the Famous Forty Oz books.
This trophy remained in the Emerald Palace up through the time of Jinjur's invasion. When the Scarecrow returned to take his throne back from Jinjur, he and his friends found themselves trapped in the palace. To escape, they decided to make a flying machine from what they could find in the palace.
Tip and the Sawhorse brought a sofa (and later fetched a matching one), the Scarecrow brought clothesline, the Tin Woodman brought palm fronds, Jack brought a broom, and the Woggle-Bug brought the gump's head. The Tin Woodman put all these pieces together into an odd flying machine that used the sofas for a body, the broom for a tail, the fronds for wings, and the Gump's head for a head. Tip brought it to life with the Powder of Life, but he didn't have enough for the sofa legs. But as they needed to fly, not walk, they decided it would be all right.
When the creature was brought to life, it immediately flew into the air, but Tip called it back. It remembered being in the forest and hearing a loud noise. Then he was revived as their flying machine. This establishes that the Powder of Life can restore dead matter to life, complete with personality and memories intact. But it must be noted that the Gump was stuffed and had no internal organs. What effect it would have on a dead body with all of its organs still inside was not explored in the Famous Forty. (There is very much an in-canon way to do zombies in Oz.)
The creature was at first called a "thing," but soon was just called "The Gump," even though just part of it was a Gump. (Okay, actually part of it was part of a Gump.) The entire company climbed inside and set off to Glinda's palace. However, it soon became night and the Gump missed Glinda's palace altogether. It crash-landed in a Jackdaw's nest, where it was quite damaged. After an attack by the Jackdaws (which the Gump managed to flop its remains and scare some off), the Woggle-Bug was able to restore it with a Wishing Pill.
After the restoration, the Gump flew back to Oz. They discovered that they were over the Munchkin Country, and then headed on to Glinda's palace. The Gump later flew the Scarecrow and his friends back to the Emerald City instead of them marching with Glinda's guards. Later, it followed Glinda and the Sawhorse as they chased Mombi.
At the climax of The Marvelous Land of Oz, the Gump is used to invade the Emerald City so Jinjur can be captured and Ozma can take the throne. As its reward, it asks to be dismantled and have its restored to its place on the wall.
According to the Famous Forty Oz books, that is where it stays. Dorothy meets it in Dorothy and the Wizard in Oz, and it tells her that it doesn't speak often, and that Ozma doesn't like it to speak much. I suppose having such a character around (a head mounted on a wall) would be unsettling for anyone.
In Baum's Queer Visitors from the Marvelous Land of Oz, the Gump is still a flying machine and it carries the Scarecrow, Tin Woodman, the Sawhorse, Jack Pumpkinhead and the Woggle-Bug to America and serves as their official means of transport. In one story, the Gump races Santa Claus' reindeer and loses to them. But overall, the Gump is a very minor character in these stories, which are of debatable continuity.
So yes, somewhere in the Emerald Palace is a living head of an animal mounted on a wall. Quite odd indeed.