Yes, that may sound like an awfully silly question. "Of course Herman Melville was a novelist!" you might be thinking. "What else could he have been?"
Please bear with me. Today, thanks partly to Melville, we're accustomed to thinking of the novel as a highly varied form. In Search of Lost Time, Ulysses, Life: A User's Manual, and Gilead are all tossed into the "novel" category. But the essence of the form, its prime examples, are the 19th-century mirrors of society -- books like Middlemarch, Anna Karenina, and Uncle Tom's Cabin. These focused on realistic settings and plots, with stories driven by the interactions of fully developed characters.
We can look at Moby-Dick and find evidence of the foregoing items: realistic settings? yes; realistic plot? for the most part, yes; character-driven story? to a certain extent. The answers need to be qualified because so much of Moby-Dick is not a novel. Great stretches of it more closely resemble philosophy or natural history or even lexicography. Moby-Dick the novel becomes engulfed in a kind of Reader's Guide to Moby-Dick.
In fact, I would argue that Melville never wrote anything that fits neatly in the novel class. His first two books, Typee and its sequel, Omoo, were openly autobiographical (though scholars have since detected ways in which Typee uses published materials to embellish what Melville actually experienced). His fourth and fifth books, Redburn and White Jacket, were lightly fictionalized autobiographical accounts. In all four books, the plots are not much beyond "what happened to me when I lived with the headhunters/bummed around the South Pacific/first sailed in a merchantman/sailed in a U.S. frigate."
Whenever Melville attempted to steer away from retellings of his own experiences, he could not bring himself to stick with the traditional novel form. Mardi starts out comfortably enough as a sea story that brings Joseph Conrad to the modern reader's mind. But somewhere in the middle, Melville abandons the blueprint he's been following and starts on an entirely different structure, giving us a series of social-political satires in the form of allegories. Moby-Dick, while much more consistent and carefully planned than Mardi, nevertheless involves a similar massive non-novelistic superstructure resting upon a straightforward adventure story. After Moby-Dick came Pierre, in which Melville's dissatisfaction with the traditional novel completely boils over. What begins as a gothic romance in the style of Anne Radcliffe and Charles Brockden Brown turns into a homicidal (and suicidal) attack on popular literature, the publishing industry, and Pierre itself.
Melville's remaining "novels" don't strengthen the case for Melville = novelist. The novelistic skeleton of Israel Potter was largely lifted from another source. The Confidence-Man, like the second half of Mardi, is a series of vignettes rather than a novel. Finally, there's Billy Budd, which is barely long enough to qualify as a "real" novel.
One view is that Melville was, at heart, a poet. He certainly had the talent. But he also had a critic's drive to pick apart and reinterpret the world around him. In that regard, he would have fit well with the great Victorian social critics, Carlyle, Macaulay, Arnold, and others. Sadly, his lack of formal education unfitted him for that role. He suffered as he struggled to find a mode of expression that would be both artistically satisfying and commercially successful. That struggle produced at least one masterpiece and decades of agony.