Theoretical studies have proven that the Hilbert space has remarkable performance in many fields of applications. Frames in tensor product of Hilbert spaces were introduced to generalize the inner product to high-order tensors. However, these techniques require tensor decomposition which could lead to the loss of information and it is a NP-hard problem to determine the rank of tensors. Here, we present a new framework, namely matrix Hilbert space to perform a matrix inner product space when data observations are represented as matrices. We preserve the structure of initial data and multi-way correlation among them is captured in the process. In addition, we extend the reproducing kernel Hilbert space (RKHS) to reproducing kernel matrix Hilbert space (RKMHS) and propose an equivalent condition of the space uses of the certain kernel function. A new family of kernels is introduced in our framework to apply the classifier of Support Tensor Machine(STM) and comparative experiments are performed on a number of real-world datasets to support our contributions.