15.2.2 Term weighting

• vector space $B$N3FMWAG$r$I$N$h$&$JCM$K$9$l$P$$$$$N$+(B?
• 3$B$D$N(B frequency (Table 15.3)
  • term frequency $\rightarrow$ t_i,j (d_j$BCf$N(Bw_i$B$N?t(B)
  • document frequency $\rightarrow$ df_i (w_i$B$,4^$^$l$k(B document$B?t(B)
  • collection frequency $\rightarrow$ cf_i (w_i$B$NAm?t(B)
  
  
  $$\begin{eqnarray*}$\sum_{j} tf_{i,j} = cf_i$ \end{eqnarray*}$$
  
  
• $B4K>W4X?t(B
  $B?M4V$N463P$O(B log $B$K6a$$(B, $B=EMWEY$O(B $B4K>W4X?t(B f(tf) $B$GI=8=$5$l$k$3(B $B$H$,B?$$(B
  
  $$\begin{eqnarray*}f(tf) &=& \sqrt{tf}\\ f(tf) &=& 1+\log(tf) \end{eqnarray*}$$
  
  
• document frequency
  • $B0UL#E*$K=EMW$J8l(B(keyword)$B$O(B1$B$D$N(B document $B=8Cf$7$d$9$$(B $\rightarrow$ df_i $B>.(B
  • $B0UL#E*$K=EMW$G$J$$8l(B(function word $B$J$I(B)$B$O6Q0l$KJ,I[(B $\rightarrow$ df_i $BBg(B
  • Table 15.4 $\rightarrow$ insurance $B$N$[$&$,(B try $B$h$j=EMW8l(B
• tf $\cdot$ idf
  • inversed document frequency (idf)
    $idf = \log(\frac{N}{df_i})$, (N $B$O(B document $B?t(B)
    w_i $B$,$9$Y$F$N(B 1 $B$D$N(B document $B$K4^$^$l$k(B $\rightarrow$ $idf = \log N$
    w_i $B$,$9$Y$F$N(B document $B$K4^$^$l$k(B $\rightarrow$ idf =0
  • term frequency $B$H(B document frequency $B$rAH$_9g$o$;$k(B
    
    $$\begin{eqnarray*}weigth(i,j) = \left\{ \begin{array}{rll} (1+\log(tf_{i.j}))... ...\geq 1\\ 0 & &if\ tf_{i,j} = 0 \end{array} \right. \nonumber \end{eqnarray*}$$
    
    
• $B$$$m$$$m$J(B tf $\cdot$ idf $\rightarrow$ Table 15.5