The advantage of CRF is that CRF resolve the label bias problem which can be happened in the MEMM model by global normalization.
Conditional Random Fields Probabilistic Models for Segmenting and Labeling Sequence Data
“This per-state normalization of transition scores implies a “conservation of score mass” (Bottou,
1991) whereby all the mass that arrives at a state must be distributed among the possible successor states. An observation can affect which destination states get the mass, but not how much total mass to pass on. This causes a bias toward states with fewer outgoing transitions. In the extreme case, a state with a single outgoing transition effectively ignores the observation. In those cases, unlike in HMMs, Viterbi decoding cannot downgrade a branch based on observations after the branch point, and models with statetransition
structures that have sparsely connected chains of states are not properly handled. The Markovian assumptions
in MEMMs and similar state-conditional models insulate decisions at one state from future decisions in a way
that does not match the actual dependencies between consecutive states.”
Label Bias Problem
Conditional Random Fields Probabilistic Models for Segmenting and Labeling Sequence Data
HMM is a useful model with a long history which has been used in many domains.
MEMM is a new model whih is inspired with HMM and Maximum Entropy theory.This model is more feasible than HMM.It can incorporate many features easily.The HMM also can incorporate features, but the processing is very strange and difficult to operate.
MEMM focuses on p（state|observation, while HMM focuses on p（observation|state）.
Maximum Entropy Markov Models for Information Extraction and Segmentation
hmm-memm-crf
A useful web site:http://homepages.inf.ed.ac.uk/lzhang10/maxent.html
A Maximum Entropy Approach to Natural Language Processing
Lagrangian duality and algorithms for the Lagrangian dual problem
The EM algorithm is a general model for maximum-likelihood estimation estimation where the data are “incomplete” or the likelihood function involes latent varibles.
The Basic Model of EM Algorithm
A Note on the Expectation-Maximization (EM) Algorithm
The lagrange multiplier method is a useful approach to solve the optimal problem under several limited conditions.
The attachment is in Chinese.
Thanks for the web site of http://www.survivor99.com/entropy/zxw/C12a.htm .
The lagrange Multiplier Method
KL divergence:
sum[p（x）log(p(x)/q(x))]
And,sum[p（x）log(p(x)/q(x))]>=0, please see the attachment for the proof.
The proof of why the KL divergence is not smaller than zero
http://www.freezq.cn/article/455.htm
The above web page is not sufficient to understand how to use this software.
Please refer to the file ‘WinRAR.chm’ for more information which will appear in your installation directory when you install the WinRAR software.
